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Abstract 


This  report  presents  a  conceptual  framework  for  multi-level  modeling  of  complex  socio- 
technical  systems,  provides  linkages  to  the  historical  roots  and  technical  underpinnings  of  this 
framework,  and  outlines  a  catalog  of  component  models  for  populating  multi-level  models.  This 
includes  a  description  of  the  "systems  movement,"  a  summary  of  philosophical  underpinnings, 
a  review  of  seminal  concepts,  an  overview  of  complex  systems,  discussion  of  complex  adaptive 
systems,  and  contrasts  of  a  range  of  systems  approaches.  Alternative  modeling  frameworks, 
including  multi-level  modeling  frameworks,  problem  structuring  methods,  and  computational 
representations,  are  also  addressed.  A  proposed  framework  is  presented  for  multi-level 
modeling  of  socio-technical  systems,  including  discussion  of  the  phenomena  typically 
associated  with  each  level,  as  well  as  a  wide  range  of  models  of  human  behavior  and 
performance.  A  comparison  is  provided  of  multi-level  representations  of  the  domains  of 
healthcare  delivery,  energy  consumption,  and  military  operations.  An  illustrative  example  is 
presented  focused  on  counterfeit  parts  in  the  military  supply  chain,  in  terms  of  both  the 
consequences  of  such  parts  and  interdicting  the  motivations  to  counterfeit.  Finally,  a  wide 
range  of  fundamental  research  issues  underlying  multi-level  modeling  of  complex  systems  is 
summarized. 
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1.0  Introduction 


Socio-technical  systems  involve  behavioral  and  social  aspects  of  people  and  society  that 
interact  with  technical  aspects  of  organizational  structure  and  processes  --  both  engineered  and 
natural  --  to  create  organizational  outcomes  and  overall  system  performance.  These  types  of 
systems  are  often  characterized  as  complex  adaptive  systems  where  independent  agents 
pursue  their  individual  objectives  while  learning  and  adapting  to  evolving  system  structures  and 
behaviors. 

Such  systems  can  be  described  at  various  levels  of  abstraction  and  aggregation.  Levels  of 
abstraction  might  vary  from  individual  human  activities,  to  processes  that  support  activities,  to 
organizations  that  invest  in  and  maintain  processes,  to  social  systems  that  create  and  regulate 
the  policy  environment  in  which  organizations  operate.  Levels  of  aggregation  could  range,  for 
example,  from  individual  humans,  to  cohorts  of  similar  humans  (e.g.,  those  with  high  risk  of 
diabetes),  to  broader  classes  of  humans  (e.g.,  economic  classes),  to  entire  populations  (e.g., 
countries'  populations). 

The  appropriate  level  at  which  to  represent  socio-technical  systems  depends  on  the  questions 
or  problems  being  addressed.  In  other  words,  the  choice  of  levels  should  be  based  on  the 
utility  of  the  representation  for  addressing  the  issues  at  hand,  rather  than  the  notion  of  their 
being  a  one,  true  "correct"  representation.  Further,  it  is  quite  possible  that  the  issues  of 
interest  will  dictate  representing  the  system  at  multiple  points  in  the  abstraction-aggregation 
space.  For  example,  one  might  need  low  abstraction,  low  aggregation  representation  of 
individual  patients  and  their  chronic  diseases,  as  well  as  a  high  abstraction,  high  aggregation 
representation  of  the  evolution  of  the  inflation  rate  for  the  costs  of  healthcare.  One 
complication  here  is  that  not  all  questions  or  problems  may  be  known  at  the  time  of  model 
conception,  since  stakeholders  may  develop  new  questions  during  the  course  of  the  model 
lifecycle. 

Thus,  multi-level  representations  are  often  needed  to  capture  the  phenomena  associated  with 
the  question  or  problem  of  interest.  This  need  often  reflects  both  the  "physics"  of  the 
phenomena  of  interest  and  the  "psychics"  of  the  solution  development  and  deployment.  More 
specifically,  the  behavioral  and  social  nature  of  stakeholders'  involvement  with  the  evolution  of 
a  solution  often  requires  some  way  for  technically  less  sophisticated  stakeholders  to  obtain  a 
deep  appreciation  for  the  phenomena  under  study.  Such  appreciation  is  often  a  prerequisite  to 
these  stakeholders  committing  to  deploy  a  solution.  Multi-level  models,  with  rich  interactive 
visualization  capabilities  can  provide  a  means  to  achieve  these  ends. 

The  ideas  summarized  in  the  preceding  paragraphs  have  been  germinating  and  evolving  for 
many  decades.  Many  people  and  disciplines  have  contributed  to  a  large  knowledge  base 
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underpinning  these  ideas.  The  primary  goal  of  this  report  is  to  present  and  summarize  this 
body  of  knowledge;  a  secondary  but  also  important  goal  is  to  illustrate  the  practical 
implications  of  drawing  upon  specific  elements  of  this  body  of  knowledge  to  address  a 
particular  problem.  These  goals  are  addressed  as  follows. 

Section  2  provides  background  for  later  sections.  It  includes  a  description  of  the  "systems 
movement,"  a  summary  of  philosophical  underpinnings,  a  review  of  seminal  concepts,  an 
overview  of  complex  systems,  discussion  of  complex  adaptive  systems,  and  contrasts  of  a  range 
of  systems  approaches.  Section  3  focuses  on  alternative  modeling  frameworks,  including  multi¬ 
level  modeling  frameworks,  problem  structuring  methods,  and  computational  representations. 

Section  4  outlines  a  proposed  framework  for  multi-level  modeling  of  socio-technical  systems, 
including  discussion  of  phenomena  typically  associated  with  each  level,  as  well  as  a  wide  range 
of  models  of  human  behavior  and  performance.  Section  5  gives  a  comparison  of  multi-level 
representations  of  the  domains  of  healthcare  delivery,  energy  consumption,  and  military 
operations.  Section  6  presents  a  detailed  example  focused  on  the  appearance  of  counterfeit 
parts  in  the  military  supply  chain,  addressing  both  the  consequences  of  such  parts  in  the  supply 
chain  and  ways  for  interdicting  the  temptations  to  counterfeit. 

Section  7  summarizes  a  wide  range  of  fundamental  research  issues  underlying  multi-level 
modeling  of  complex  systems.  Section  8  summarizes  the  findings  and  implications  of  this 
report.  A  comprehensive  list  of  references  is  provided  at  the  end  of  this  report. 

The  overarching  objective  of  this  report  is  to  provide  a  rigorous  foundation  for  multi-level 
modeling  to  support  decision  making  in  complex  socio-technical  systems.  These  models  are 
envisioned  as  being  created  to  support  explorations  of  answers  to  decision  makers'  questions, 
ranging  from  strategic  investments  to  system  design  and  development  to  operation  and 
maintenance  of  complex  systems.  Targeted  domains  include  military  operations,  urban 
resilience,  energy  consumption  and  healthcare  delivery. 

2.0  Background 


2.1  Systems  Movement 

The  systems  movement  emerged  from  the  formalization  of  systems  theory  as  an  area  of  study 
during  and  following  World  War  II,  although  it  can  be  argued  that  the  physicists  and  chemists  of 
the  19*  Century  contributed  to  the  foundations  of  this  area.  Before  delving  into  the  ideas 
emerging  in  the  1940s  and  beyond,  it  is  important  to  distinguish  four  aspects  of  the  systems 
movement: 
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Systems  Thinking  is  the  process  of  understanding  how  things  influence  one  another  within  a 
whole  and  represents  an  approach  to  problem  solving  that  views  "problems"  as  components  of 
an  overall  system 

Systems  Philosophy  is  the  study  of  systems,  with  an  emphasis  on  causality  and  design.  The  most 
fundamental  property  of  any  system  is  the  arbitrary  boundary  that  humans  create  to  suit  their 
own  purposes 

Systems  Science  is  an  interdisciplinary  field  that  studies  the  nature  of  complex  systems  in 
nature  and  society,  to  develop  interdisciplinary  foundations,  which  are  applicable  in  a  variety  of 
areas,  such  as  engineering,  biology,  medicine  and  economics 

Systems  Engineering  is  an  interdisciplinary  field  focused  on  identifying  how  complex 
engineering  undertakings  should  be  designed,  developed  and  managed  over  their  life  cycles 
Contrasting  these  four  aspects  of  systems,  it  is  important  to  recognize  that  different  disciplines 
tend  to  see  "systems"  quite  differently,  for  the  most  part  due  to  the  varying  contexts  of  interest 
(Adams,  et  al.,  2013).  Thus,  a  systems  scientist  studying  marsh  ecosystems  and  a  systems 
engineer  designing  and  developing  the  next  fighter  aircraft  will,  from  a  practical  perspective  at 
least,  have  much  less  in  common  than  the  term  "system"  might  lead  one  to  expect.  The  key 
point  is  that  systems  exist  in  contexts  and  different  contexts  may  (and  do)  involve  quite 
disparate  phenomena. 


2.2  Philosophical  Background 

There  are  many  interpretations  of  what  system  thinking  means  and  the  nature  of  systems 
thinkers.  Some  are  inclined  towards  model-based  deduction,  while  others  are  oriented 
towards  data-driven  inference.  The  former  extol  the  deductive  powers  of  Newton  and  Einstein, 
while  the  latter  are  enamored  with  the  inferential  capabilities  of  Darwin.  These  different 
perspectives  reflect  different  epistemologies. 

The  study  of  epistemology  involves  the  questions  of  what  is  knowledge,  how  can  it  be  acquired, 
and  what  can  be  known.  The  empiricism  branch  of  epistemology  emphasizes  the  value  of 
experience.  The  idealism  branch  sees  knowledge  as  innate.  The  rationalism  branch  relies  on 
reason.  The  constructivism  branch  seeks  knowledge  in  terms  of  creation.  These  branches 
differ  in  terms  of  how  they  represent  knowledge,  in  particular  how  this  knowledge  is  best 
modeled  and  simulated  (Tolk,  2013). 

There  are  many  possible  views  of  complexity  and  complex  systems  (Rouse,  2007).  Systems 
paradigms  for  representation  of  knowledge  include  hierarchical  mappings,  state  equations, 
nonlinear  mechanisms,  and  autonomous  agents  (Rouse,  2003).  For  hierarchical  mappings, 
complexity  is  typically  due  to  large  numbers  of  interacting  elements.  With  uncertain  state 
equations,  complexity  is  due  to  large  numbers  of  interacting  state  variables  and  significant 
levels  of  uncertainty.  Discontinuous,  nonlinear  mechanisms  attribute  complexity  to  departures 
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from  the  expectations  stemming  from  continuous,  linear  phenomena.  Finally,  autonomous 
agents  generate  complexity  via  the  reactions  of  agents  to  each  other's  behavior  and  lead  to 
emergent  phenomena.  The  most  appropriate  choice  among  these  representations  depends  on 
how  the  boundaries  of  the  system  of  interest  are  defined  (Robinson,  et  al.,  2011). 

Horst  Rittel  argued  that  the  choice  of  representation  is  particularly  difficult  for  "wicked 
problems"  (Rittel  &  Webber,  1973).  There  is  no  definitive  formulation  of  a  wicked  problem. 
Wicked  problems  have  no  stopping  rule  -  there  is  always  a  better  solution,  e.g.,  "fair"  taxation 
and  "just"  legal  systems.  Solutions  to  wicked  problems  are  not  true  or  false,  but  good  or  bad. 
There  is  no  immediate  nor  ultimate  test  of  a  solution  to  a  wicked  problem.  Wicked  problems 
are  not  amenable  to  trial  and  error  solutions.  There  is  no  innumerable  (or  an  exhaustively 
describable)  set  of  potential  solutions  and  permissible  operations.  Every  wicked  problem  is 
essentially  unique.  Every  wicked  problem  can  be  considered  a  symptom  of  another  problem. 
Discrepancies  in  representations  can  be  explained  in  numerous  ways  -  the  choice  of 
explanation  determines  the  nature  of  problem's  resolution.  Problem  solvers  are  liable  for  the 
consequences  of  the  actions  their  solutions  generate.  Many  real  world  problems  have  the 
above  characteristics. 

The  notion  of  wicked  problems  raises  the  possibility  of  system  paradoxes  (Baldwin,  et  al., 
2010).  Classic  paradoxes  include  whether  light  is  a  particle  or  a  wave.  Contemporary 
paradoxes  include  both  collaborating  and  competing  with  the  same  organization.  The 
conjunction  paradox  relates  to  the  system  including  element  A  and  element  not  A.  The  bi¬ 
conditional  paradox  holds  if  A  implies  B  and  B  implies  A.  For  the  equivalence  paradox,  system 
elements  have  contradictory  qualities.  With  the  implication  paradox,  one  or  more  system 
elements  lead  to  its  own  contradiction.  The  disjunction  paradox  involves  systems  that  are  more 
than  the  sum  of  their  parts.  Finally,  the  perceptual  paradox  reflects  perceptions  of  a  system 
that  are  other  than  reality. 

Finally,  there  are  fundamental  theoretical  limits  as  to  what  we  can  know  about  a  system  and  its 
properties  (Rouse,  1986,  1989,  1991).  There  are  limits  of  system  information  processing 
capabilities  (Chaitin,  1974),  limits  to  identifying  signal  processing  and  symbol  processing 
models,  limits  of  validating  knowledge  bases  underlying  intelligent  systems,  and  limits  of 
accessibility  of  mental  models  in  terms  of  forms  and  content  of  representations.  The 
implication  is  that  models  are  inherently  approximations  of  reality  and  may  be  biased  and 
limited  in  significant  ways. 


2.3  Seminal  Concepts 


2.3.1  Systems  Science 

The  experiences  of  the  problem-driven  research  in  World  War  II  led  many  now-notable 
researchers  to  develop  new  concepts,  principles,  models,  methods  and  tools  for  specific 
military  problems  that  they  later  generalized  to  broader  classes  of  phenomena.  The  systems 
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theorists  included  Norbert  Weiner  (1948)  who  generalized  control  theory  into  the  concept  of 
cybernetics.  Weiner  defined  cybernetics  as  the  study  of  control  and  communication  in  the 
animal  and  the  machine.  Studies  in  this  area  focus  on  understanding  and  defining  the  functions 
and  processes  of  systems  that  have  goals  and  that  participate  in  circular,  causal  chains  that 
move  from  action  to  sensing  to  comparison  with  desired  goal,  and  back  again  to  action. 
Concepts  studied  include,  but  are  not  limited  to,  learning,  cognition,  adaptation,  emergence, 
communication,  efficiency  and  effectiveness.  Later  extensions  of  control  theory  include  optimal 
state  filtering  (Kalman,  1960)  and  optimal  control  (Bellman,  1957;  Pontryagin,  et  al,  1962) 

Claude  Shannon  (1948)  developed  information  theory  to  address  the  engineering  problem  of 
the  transmission  of  information  over  a  noisy  channel.  The  most  important  result  of  this  theory 
is  Shannon's  coding  theorem,  which  establishes  that,  on  average,  the  number  of  bits  needed  to 
represent  the  result  of  an  uncertain  event  is  given  by  its  entropy,  where  entropy  is  a  measure  of 
the  uncertainty  associated  with  a  random  variable.  In  the  context  of  information  theory,  the 
term  refers  to  Shannon  entropy,  which  quantifies  the  expected  value  of  the  information 
contained  in  a  message,  typically  measured  in  binary  digits  or  bits.  Shannon's  noisy-channel 
coding  theorem  states  that  reliable  communication  is  possible  over  noisy  channels  provided 
that  the  rate  of  communication  is  below  a  certain  threshold,  called  the  channel  capacity.  The 
channel  capacity  can  be  approached  in  practice  by  using  appropriate  encoding  and  decoding 
systems. 

Ross  Ashby  (1951,  1956)  added  the  Law  of  Requisite  Variety  to  the  canon.  Put  succinctly,  only 
variety  can  destroy  variety.  More  specifically,  if  a  system  is  to  be  fully  regulated,  the  number  of 
states  of  its  control  mechanism  must  be  greater  than  or  equal  to  the  number  of  states  in  the 
system  being  controlled.  Thus,  in  order  for  an  enterprise  to  reduce  the  variety  manifested  by 
its  environment  to  yield  less  varied  products  and  services,  it  must  have  sufficient  variety  in  its 
business  processes. 

Bertalanffy  (1968)  developed  General  Systems  Theory  over  several  decades,  with  particular 
interest  in  biological  and  open  systems,  i.e.,  those  that  continuously  interact  with  their 
environments.  The  areas  of  systems  science  that  he  included  in  his  overall  framework 
encompass  cybernetics,  theory  of  automata,  control  theory,  information  theory,  set,  graph  and 
network  theory,  decision  and  game  theory,  modeling  and  simulation,  and  dynamical  systems 
theory  -  in  other  words,  virtually  all  of  systems  science.  Bertalanffy  includes  consideration  of 
systems  technology  including  control  technology,  automation,  computerization,  and 
communications.  Had  the  field  of  artificial  intelligence  existed  in  his  time,  that  area  would  have 
surely  been  included  as  well.  As  is  often  the  case  with  grand  generalizations,  it  is  often  difficult 
to  argue  with  the  broad  assertions  but  sometimes  not  easy  to  see  the  leverage  gained. 

Ackoff  (1981)  coined  the  term  "system  of  systems"  that  has  gained  great  currency  of  late.  He 
recognized  that  organizations  could  be  seen  as  systems.  In  this  context,  he  outlined  a 
classification  of  systems  (self-maintaining,  goal-seeking,  multi-goal  seeking,  purposive  system), 
and  elaborated  the  notions  of  system  state,  system  changes,  and  system  outcomes,  where 


FtepoitNo.  SERC-20l3-1R-a20-2 
J  une  6,  2013 

UNCLASSIFIED 

12 


Contract  Number:  H98230-08-D-0171 


TO  0029,  RT044a 


outcomes  are  seen  as  the  consequences  of  system  responses,  not  just  the  response  variables  in 
themselves.  He  further  elaborated  organizational  systems  as  being  variety-increasing  or 
variety-decreasing,  and  discusses  adaptation  and  learning. 


2.3.2  Economics/Cognition 

It  may  seem  odd  to  group  economics  with  cognition.  However,  much  seminal  thinking  arose 
from  people  who  studied  behavioral  and  social  phenomena  associated  with  economic 
processes.  Nobel  Prize  winner  Kenneth  Arrow  (1951, 1954)  developed  social  choice  theory,  the 
associated  impossibility  theorem,  equilibrium  theory,  and  the  economics  of  information.  Nobel 
Prize  winner  Herbert  Simon  (1957,  1962)  studied  bounded  rationality,  satisficing  vs.  optimizing, 
behavioral  complexity  as  a  reflecting  of  environmental  complexity,  human  information 
processing,  and  artificial  intelligence.  Nobel  Prize  winner  Daniel  Kahneman  (2011),  with  his 
colleague  Amos  Tversky,  studied  human  decision  making  biases  and  heuristics  for  several 
decades.  Finally,  George  Miller  (1956)  contributed  to  cognitive  psychology,  cognitive  science, 
and  psycholinguistics  (which  links  language  and  cognition),  and  studies  of  short-term  memory - 
coming  up  with  oft-cited  "magic  number  seven." 

This  body  of  work  provides  important  insights  into  socio-technical  systems  (as  well  as  into  how 
to  win  a  Nobel  Prize  in  Economics).  Put  simply,  the  classical  notional  of  "economic  man"  as  a 
completely  rational,  decision  maker  who  can  be  counted  on  to  make  optimal  choices  is  often  a 
wildly  idealistic  assumption.  The  phenomena  studied  by  Arrow,  Simon,  Kahneman  and  Miller 
make  classical  mathematical  economics  quite  difficult.  On  the  other  hand,  these  phenomena 
can  make  agent-based  simulations  quite  important.  Later  in  this  report,  the  modeling  of  human 
decision  making  is  considered  in  some  depth,  with  many  concepts  traceable  back  to  the  seminal 
thinkers  discussed  in  this  section. 


2.3.3  Operations  Research 

Operations  research  emerged  from  World  War  II  and  efforts  to  look  at  military  operations  and 
improve  them.  Philip  Morse  was  a  pioneer  in  the  research  philosophy  of  immersing  problem 
solvers  in  the  complex  domains  where  solutions  are  sought.  The  key  element  was  the  emphasis 
on  research  in  operational  contexts  rather  than  just  study  of  mathematical  formalisms.  Morse 
(1951,  1958)  authored  the  first  books  in  the  United  States  in  this  area,  and  went  on  to  publish 
an  award-winning  book  on  the  application  of  OR  to  libraries  (Morse,  1968). 

C.  West  Churchman  was  internationally  known  for  his  pioneering  work  in  operations  research, 
system  analysis  and  ethics.  He  was  recognized  for  his  then  radical  concept  of  incorporating 
ethical  values  into  operating  systems  (Churchman,  1971).  Ackoff  received  his  doctorate  in 
philosophy  of  science  in  1947  as  Churchman's  first  doctoral  student  (Ackoff  &  Churchman, 
1957).  He  became  one  of  the  most  important  critics  of  the  so-called  "technique-dominated 
Operations  Research",  and  proposed  more  participative  approaches.  He  argued  that  any 
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human-created  system  can  be  characterized  as  a  "purposeful  system"  when  it's  "members  are 
also  purposeful  individuals  who  intentionally  and  collectively  formulate  objectives  and  are  parts 
of  larger  purposeful  systems"  (Ackoff  &  Emery,  1972). 

More  recently.  Operations  Research  has  come  to  be  dominated  by  applied  mathematics  as  an 
end  in  itself.  The  quest  for  provably  optimal  solutions  of  problems  has  resulted  in  problems 
being  scaled  down,  often  dramatically,  to  enable  analytical  proofs  of  optimality.  The  constructs 
of  theorems  and  proofs  have  often  displaced  the  intention  to  actually  solve  realistically 
complex  problems.  The  value  of  immersing  researchers  in  complex  operational  domains  has 
often  come  to  be  discounted  as  impractical  by  the  researchers  themselves. 

2.3.4  Sociology 

Talcott  Parsons  was  one  of  the  first  social  scientists  to  become  interested  in  systems 
approaches.  He  developed  action  theory,  the  principle  of  voluntarism,  understanding  of  the 
motivation  of  social  behavior,  the  nature  of  social  evolution,  and  the  concept  of  open  systems 
(Parsons,  1937, 1951,  1956).  This  very  much  set  the  stage  for  the  emergence  of  socio-technical 
systems  as  an  area  of  study  in  its  own  right. 

The  idea  of  work  systems  and  the  socio-technical  systems  approach  to  work  design  was 
originated  by  Trist,  Emery  and  colleagues  (Trist  &  Bamforth,  1951;  Emery  &  Trist,  1965,  1973). 
This  included  research  on  participative  work  design  structures  and  self-managing  teams.  It  also 
led  to  a  deep  appreciation  of  the  roles  of  behavioral  and  social  phenomena  in  organizational 
outcomes  and  performance. 


2.4  Complex  Systems 

This  section  considers  differing  perspectives  on  the  nature  of  complex  systems;  drawing  upon 
several  recently  published  review  papers  (Rouse,  2003,  2005,  2007;  Rouse  &  Serban,  2011).  It 
is  useful  to  note  that  different  disciplines,  in  part  due  to  the  contexts  in  which  they  work,  can 
have  significantly  varying  views  of  complexity  and  complex  systems. 

Several  concepts  are  quite  basic  to  understanding  complex  systems.  One  key  concept  is  the 
dynamic  response  of  a  system  as  a  function  of  structural  and  parametric  properties  of  the 
system.  The  nature  of  the  response  of  a  system,  as  well  as  the  stability  and  controllability  of 
this  response,  is  a  central  concern.  Many  operations  research  studies  focus  on  steady-state 
behavior,  while  economics  research  addresses  equilibrium  behavior.  However,  transient 
behaviors  -  whether  of  the  weather  or  the  financial  system  -  are  often  the  most  interesting  and 
sometimes  the  most  damaging. 

Another  basic  concept  is  uncertainty  about  a  system's  state.  The  state  of  a  system  is  the 
quantities/properties  of  the  system  whose  knowledge,  along  with  future  inputs,  enables 
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prediction  of  future  values  of  this  set  of  variables.  Uncertainty  of  system  state  limits  the 
effectiveness  of  control  strategies  in  assuring  system  performance.  State  estimation  -  filtering, 
smoothing  and  prediction  -  is  an  important  mechanism  for  obtaining  the  best  information  for 
controlling  a  complex  system.  Related  topics  include  the  value  of  information  and  performance 
risks,  e.g.,  consequences  of  poor  performance. 

It  is  useful  differentiate  the  notions  of  "system"  and  "complex  system"  (Rouse,  2003).  A  system 
is  a  group  or  combination  of  interrelated,  interdependent,  or  interacting  elements  that  form  a 
collective  entity.  Elements  may  include  physical,  behavioral,  or  symbolic  entities.  Elements 
may  interact  physically,  computationally,  and/or  by  exchange  of  information.  Systems  tend  to 
have  goals/purposes,  although  in  some  cases  the  observer  ascribes  such  purposes  to  the 
system  from  the  outside  so  to  speak. 

Note  that  a  control  system  could  be  argued  to  have  elements  that  interact  computationally  in 
terms  of  feedback  control  laws,  although,  one  might  also  argue  that  the  interaction  takes  place 
in  terms  of  the  information  that  embodies  the  control  laws.  One  could  also  describe  the 
control  function  in  terms  of  physical  entities  such  as  voltages  and  displacements.  Thus,  there 
are  (at  least)  three  different  representations  of  the  same  functionality  --  hence,  the  "and/or"  in 
the  definition. 

A  complex  system  is  one  whose  perceived  complicated  behaviors  can  be  attributed  to  one  or 
more  of  the  following  characteristics:  large  numbers  of  elements,  large  numbers  of 
relationships  among  elements,  nonlinear  and  discontinuous  relationships,  and  uncertain 
characteristics  of  elements  and  relationships.  From  a  functional  perspective,  the  presence  of 
complicated  behaviors,  independent  of  underlying  structural  features,  may  be  sufficient  to 
judge  a  system  to  be  complex.  Complexity  is  perceived  because  apparent  complexity  can 
decrease  with  learning. 

More  specifically,  system  complexity  tends  to  increase  with  the  number  of  elements,  number  of 
relationships,  nature  of  relationships  (i.e.,  logical:  AND  vs.  OR  &  NAND;  functional:  linear  vs. 
nonlinear;  spatial:  lumped  vs.  distributed;  structural:  for  example,  feed  forward  vs.  feedback; 
response:  static  vs.  dynamic;  time  constant:  (not  too)  fast  vs.  (very)  slow,  and  uncertainty: 
known  properties  vs.  unknown  properties,  and  knowledge,  experience  and  skills  (relative  to  all 
of  the  above,  relative  to  observer's  intentions). 

The  issue  of  intentions  is  summarized  in  Figure  1  (Rouse,  2007).  If  one's  intention  is  simply  to 
classify  as  observed  object  as  an  airplane,  the  object  is  not  particularly  complex.  If  one  wanted 
to  explain  why  it  is  an  airplane,  the  complexity  of  an  explanation  would  certainly  be  greater 
than  that  of  a  classification.  For  these  two  intentions,  one  is  simply  describing  an  observed 
object. 
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Complexity  -  f  (Intentions) 


Input (U) 


>• 


System  (S) 


Output  (Y) 

- ► 


Intention 

Example 

Classification 

"It's  an  instance  of  type  S." 

Explanation 

"It's  type  S  because  ..." 

Prediction 

"Its  future  output  will  be  Y." 

Control 

"If  input  is  U,  its  output  will  be  Y." 

Detection 

"Its  output  is  not  Y,  but  should  be." 

Diagnosis 

"Its  output  is  not  Y  because  ..." 

Figure  1.  Relationship  of  Complexity  and  Intentions 

If  one's  intention  is  to  predict  the  future  state  of  the  airplane,  complexity  increases 
substantially  as  one  would  have  to  understand  the  dynamic  nature  of  the  object,  at  least  at  a 
functional  level  but  perhaps  also  at  a  structural  level.  Control  requires  a  higher  level  of 
knowledge  and  skill  concerning  input-output  relationships.  Intentions  related  to  detection  and 
diagnosis  require  an  even  greater  level  of  knowledge  and  skill  concerning  normal  and  off- 
normal  behaviors  in  terms  of  symptoms,  patterns,  and  structural  characteristics  of  system 
relationships.  The  overall  conclusion  is  that  the  complexity  of  a  system  cannot  be  addressed 
without  considering  the  intentions  associated  with  addressing  the  system. 

The  nature  of  human  and  social  phenomena  within  a  system  has  thus  far  not  been  considered. 
Systems  where  such  phenomena  play  substantial  roles  are  often  considered  to  belong  to  a  class 
of  systems  termed  complex  adaptive  systems  (Rouse,  2000,  2008).  Systems  of  this  type  have 
the  following  characteristics: 

•  They  tend  to  be  nonlinear,  dynamic  and  do  not  inherently  reach  fixed  equilibrium 
points.  The  resulting  system  behaviors  may  appear  to  be  random  or  chaotic. 

•  They  are  composed  of  independent  agents  whose  behavior  can  be  described  as  based 
on  physical,  psychological,  or  social  rules,  rather  than  being  completely  dictated  by  the 
physical  dynamics  of  the  system. 

•  Agents'  needs  or  desires,  reflected  in  their  rules,  are  not  homogeneous  and,  therefore, 
their  goals  and  behaviors  are  likely  to  differ  or  even  conflict  -  these  conflicts  or 
competitions  tend  to  lead  agents  to  adapt  to  each  other's  behaviors. 

•  Agents  are  intelligent  and  learn  as  they  experiment  and  gain  experience,  perhaps  via 
"meta"  rules,  and  consequently  change  behaviors.  Thus,  overall  system  properties 
inherently  change  over  time. 
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•  Adaptation  and  learning  tends  to  result  in  self-organization  and  patterns  of  behavior 
that  emerge  rather  than  being  designed  into  the  system.  The  nature  of  such  emergent 
behaviors  may  range  from  valuable  innovations  to  unfortunate  accidents. 

•  There  is  no  single  point(s)  of  control  -  system  behaviors  are  often  unpredictable  and 
uncontrollable,  and  no  one  is  "in  charge."  Consequently,  the  behaviors  of  complex 
adaptive  systems  usually  can  be  influenced  more  than  they  can  be  controlled. 

As  might  be  expected,  multi-level  modeling  of  complex  socio-technical  systems  having  these 
characteristics  creates  significant  complications.  For  example,  the  simulation  of  such  models 
often  does  not  yield  the  same  results  each  time.  Random  variation  may  lead  to  varying  "tipping 
points"  among  stakeholders  for  different  simulation  runs.  These  models  can  be  useful  in  the 
exploration  of  leading  indicators  of  the  different  tipping  points  and  in  assessing  potential 
mitigations  for  undesirable  outcomes.  This  topic  is  addressed  in  more  detail  later. 

Snowden  and  Boone  (2007)  have  argued  that  there  are  important  distinctions  that  go  beyond 
complex  systems  versus  complex  adaptive  systems.  Their  Cynefin  Framework  includes  simple, 
complicated,  complex  and  chaotic  systems.  Simple  systems  can  be  addressed  with  best 
practices.  Complicated  systems  are  the  realm  of  experts.  Complex  systems  represent  the 
domain  of  emergence.  Finally,  chaotic  systems  require  rapid  responses  to  stabilize  potential 
negative  consequences.  The  key  distinction  with  regard  to  the  types  of  contexts  discussed  in 
this  report  is  complex  versus  complicated  systems.  There  is  a  tendency,  they  contend,  for 
experts  in  complicated  systems  to  perceive  that  their  expertise,  methods  and  tools  are  much 
more  applicable  to  complex  systems  than  is  generally  warranted. 


2.5  Systems  Approaches 

The  evolution  of  systems  practice  has  a  rich  history.  During  the  1900-1920s,  Henry  Gantt 
(1861-1919),  Frederick  Taylor  (1856-1919),  and  Frank  Gilbreth  (1868-1924)  pioneered  scientific 
management.  Quality  assurance  and  quality  control  emerged  in  the  1920-30s,  led  by  Walter 
Shewhart  (1891-1967).  Peter  Drucker  (1909-2005)  and  Chester  Barnard  (1886-1961)  formalized 
corporate  operations  management  in  the  1940-50s.  During  and  following  World  War  II,  Philip 
Morse  (1903-1985),  C.  West  Churchman  (1913-2004),  George  Dantzig  (1914-2005),  and  Russell 
Ackoff  (1919-2009)  were  leading  thinkers  in  operations  research.  Stafford  Beer  (1926-2002) 
articulated  the  foundations  of  management  cybernetics  in  the  1960-70s.  W.  Edwards  Deming 
(1900-1993)  and  Joseph  Juran  (1904-2008)  brought  total  quality  management  to  the  U.S.  in  the 
1970-80S.  Michael  Hammer  (1948-2008)  and  James  Champy  led  the  wave  of  business  process 
reengineering  in  the  1990s.  Taiichi  Ohno's  (1912-1990)  innovations  in  six  sigma  and  lean 
production  gained  traction  in  the  U.S.  in  the  1990-2000s.  Most  recently,  Daniel  Kahneman  has 
led  the  way  for  behavioral  economics  in  the  2010s. 

Over  more  than  a  century,  systems  thinking  tried  to  become  increasingly  rigorous,  focusing  on 
mathematics,  statistics,  and  computation.  During  the  1960-70s,  many  thought  leaders  began  to 
recognize  that  forcing  all  phenomena  into  this  mold  tended  to  result  in  many  central 
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phenomena  being  assumed  away  to  allow  for  the  much-sought  theorems  and  proofs  to  be 
obtained.  In  particular,  behavioral  and  social  phenomena  associated  with  complex  systems 
were  simplified  by  viewing  humans  as  constrained  but  rational  decision  makers  who  always 
made  choices  that  optimized  the  objective  performance  criteria  (linear  if  lucky). 

The  reaction,  particularly  in  the  United  Kingdom,  to  such  obviously  tenuous  assumptions  was 
the  emergence  of  the  notion  of  hard  vs.  soft  systems  thinking  (Pidd,  2004).  Table  1  contrasts 
these  two  points  of  view.  Hard  systems  thinking  seeks  quantitative  solutions  of  mathematical 
models  that  are  assumed  to  be  valid  representations  of  the  real  world  and,  consequently,  will 
inherently  be  embraced  once  they  are  calculated.  Soft  systems  thinking  sees  modeling  as  a 
means  for  exploration  and  learning  via  intellectual  and  inherently  approximate  constructs  open 
to  discussion  and  debate. 


Hard  Systems  Thinking 

Soft  Systems  Thinking 

Oriented  to  goal  seeking 

Oriented  to  learning 

Assumes  the  world  contains  systems  that  can  be 
"engineered" 

Assumes  the  world  is  problematical  but  can  be  explored 
using  models  or  purposeful  activity 

Assumes  systems  models  to  be  models  of  the 
world 

Assumes  systems  models  to  be  intellectual  constructs  to 
help  debate 

Talks  the  language  of  problems  and  solutions 

Talks  the  language  of  issues  and  accommodations 

Philosophically  positivistic 

Philosophically  phenomenological 

Sociologically  functionalist 

Sociologically  interpretative 

Systematicity  lies  in  the  world 

Systematicity  lies  in  the  process  of  inquiry  into  the  world 

Table  1.  Hard  vs  Soft  Systems  Thinking  (Pidd,  2004) 


Table  2  contrasts  systems  approaches  (Jackson,  2003).  Hard  systems  thinking  represents  but 
one  cell  in  this  table.  Other  methods  are  much  less  "closed  form"  in  orientation,  relying  more 
on  simulation  as  well  as  participative  mechanisms.  The  keys  for  these  latter  mechanisms  are 
insights  and  consensus  building. 
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Participants 

Unitary 

Pluralist 

Coercive 

Systems 

Simple 

Hard  Systems  Thinking 

Soft  Systems 

Approaches 

Emancipatory  Systems 
Thinking 

Complex 

System  Dynamics 

Organizational  Cybernetics 

Complexity  Theory 

Postmodern  Systems 
Thinking 

Table  2.  Systems  Approaches  (Jackson,  2003) 


Table  3  contrasts  methodologies  and  problems  (Jackson  &  Keys,  1984).  Again,  only  one  cell  of 
the  table  includes  traditional  operations  research  and  systems  analysis.  For  other  than 
mechanical  problems  with  a  single  decision  maker,  much  more  participative  approaches  are 
warranted,  at  least  if  the  goal  is  solving  the  problem  of  interest  rather  than  just  modeling  the 
"physics"  of  the  context. 


Mechanical 

Systemic 

Unitary  -  One  Decision  Maker 

Operations  Research 

Systems  Engineering 

Systems  Analysis 

Organizational  Cybernetics  Socio- 
Technical  Systems 

Pluralist  -  Multiple 

Independent  Decision  Makers 

Singerian  Inquiry  Systems 

Strategic  Assumption  Methods 

Wicked  Problem  Formulations 

General  Systems  Theory 

Complex  Adaptive  Systems 

Soft  Systems  Methodology 

Table  3.  Methodologies  vs.  Problems  (Jackson  &  Keys,  1984) 


Table  4  summarizes  Ulrich's  (2008)  levels  of  system  practice.  He  differentiates  hard  versus  soft 
in  terms  of  three  categories  -  one  hard  and  two  versions  of  soft.  One  class  of  soft  management 
addresses  change  while  the  other  addresses  conflict.  The  key  disciplines  and  tools  vary 
substantially  across  these  three  categories. 
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Aspect 

Operational  Systems 
Management 

Strategic  Systems 
Management 

Normative  Systems 
Management 

Dominating 

Interpretation 

Systematic 

Systemic 

Critical  Idea  of  Reason 

Strand  of  Systems 
Thinking 

Hard  -  Mechanistic 

Paradigm 

Soft  -  Evolutionary 

Paradigm 

Soft  -  Normative 

Paradigm 

Dimension  of 

Rationalization 

Instrumental 

Strategic 

Communicative 

Main  Object  of 
Rationalization 

Resources  -  Means  of 

Production 

Policies -Steering 

Principles 

Norms  -  Collective 

Preferences 

Task  of  the  Expert 

Management  of 

Scarceness 

Management  of 

Complexity 

Management  of  Conflict 

Type  of  Pressure 

Costs 

Change 

Conflict 

Basic  Approach 

Optimization 

Steering  Capacity 

Consensus 

Goodness  Criterion 

Efficient 

Effective 

Ethical 

Theory-Practice 

Mediation 

Decisionistic 

Technocratic 

Pragmatistic 

Key  Disciplines 

Decision  Theory, 

Economics,  Engineering 

Game  Theory,  Ecology, 

Social  Sciences 

Discourse  Theory,  Ethics, 
Critical  Theory 

Example  Tools 

Cost-Benefit  Analysis, 

Linear  Optimization 

Sensitivity  Analysis,  Large- 
Scale  Simulation 

Systems  Assessment, 

Ideal  Planning 

Trap  to  Avoid 

Suboptimization 

Social  Technology 

Excluding  the  Affected 

Table  4.  Levels  of  Systems  Practice  (Ulrich,  2008) 


Table  5  summarizes  Jackson's  (2003)  Critical  Systems  Practice.  The  most  important  aspect  of 
his  guidance  is  to  remain  open  to  the  range  of  possibilities  in  Tables  1-4.  From  the  perspective 
of  multi-level  modeling  of  complex  systems,  this  means  that  the  nature  of  the  levels  and  how 
they  are  populated  with  component  models  should  be  driven  by  the  issues  of  interest,  the 
phenomena  underlying  these  issues,  and  the  orientations  of  the  key  stakeholders  in  the 
problem  framing  and  solving  processes. 
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Creativity 

Task 

To  highlight  significant  concerns,  issues  and  problems 

Tools 

Creativity-enhancing  devices  employing  multiple  perspectives 

Outcome 

Dominant  and  dependent  concerns,  issues  and  problems 

Choice 

Task 

To  choose  an  appropriate  generic  systems  methodology 

Tools 

Methods  for  revealing  methodological  strengths  and  weaknesses 

Outcome 

Dominant  and  dependent  generic  systems  methodologies 

Implementation 

Task 

To  arrive  at  and  implement  specific  positive  change  proposals 

Tools 

Generic  systems  methodologies 

Outcome 

Highly  relevant  and  coordinated  change  yielding  improvements 

Reflection 

Task 

To  produce  learning  about  the  problem  and  solution 

Tools 

Clear  understanding  about  the  current  state  of  knowledge 

Outcome 

Research  findings  that  fed  back  into  practice 

Table  5.  Critical  Systems  Practice  (Jackson,  2003) 


Pidd  (2004)  offers  the  notion  of  complementarity  as  a  way  of  rationalizing  the  relationship 
between  hard  and  soft  approaches.  He  argues  that  hard  and  soft  approaches  are 
complementary  to  each  other,  but  their  complementarity  is  asymmetric.  He  asserts  that  any 
problem  situation  in  human  affairs  will  always  at  some  level  entail  differences  in  world  views 
that  the  "soft"  approaches  can  be  used  to  explore.  Within  that  exploration,  any  or  all  of  the 
hard  approaches  can  be  adopted  as  a  conscious  strategy.  The  reverse  strategy  is  not  available 
because  it  entails  abandoning  the  ontological  stance  of  hard  approaches.  In  other  words,  hard 
approaches  are  often  inextricably  tied  to  paradigms  and  assumptions  that  are  central  to  their 
problem  solving  power. 

Gharajedaghi  (2011)  articulates  a  system  methodology  for  supporting  complex  adaptive 
systems.  The  methodology  focuses  on  functions,  structure,  and  processes.  To  define  functions, 
he  argues  that  one  should  clarify  which  products  solve  which  problems  for  which  customers. 
To  define  structure,  he  advances  the  idea  of  a  modular  design  that  defines  complementary 
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relationships  among  relatively  autonomous  units.  Finally,  design  of  processes  involves  using  a 
multidimensional  modular  design  based  on  the  triplet  input  (technology),  output  (products), 
and  environments  (markets). 

This  brief  discussion  of  systems  approaches  serves  to  set  the  stage  for  alternative  approaches 
to  multi-level  modeling  of  complex  socio-technical  systems.  The  nature  of  these  systems 
usually  precludes  fully  modeling  them  with  first-principles  physics  models.  Socio-technical 
systems  are,  by  no  means,  as  mechanistic  and  predictable  as  purely  physical  systems  like 
bouncing  balls  or  gear  trains.  Yet,  there  are  well-developed  approaches  for  addressing  problem 
solving  in  complex  socio-technical  systems.  Valid  predictions,  and  occasionally  optimization, 
are  certainly  of  interest.  However,  insights  into  phenomena,  sensitivities  to  key  parameters, 
and  consensus  building  are  often  the  overarching  goals. 

3.0  Alternative  Frameworks 


This  report  is  focused  on  multi-level  modeling  of  complex  systems.  The  idea  of  representing 
systems  at  multiple  levels  of  abstraction  and  aggregation  is  certainly  far  from  novel 
(Rasmussen,  1986,  1994).  Differing  levels  of  abstraction  enable  representation  of  seemingly 
disparate  phenomena,  e.g.,  healthcare  cost  reimbursement  policies  versus  impact  of  exercise 
on  blood  pressure  levels.  Differing  levels  of  aggregation  allow  consideration  of  varying  levels  of 
detail,  perhaps  at  each  level  of  abstraction.  Thus,  an  individual  patient's  blood  pressure  is 
important  to  predicting  their  risks  of  chronic  diseases,  but  each  person's  consumption 
inclinations  need  not  be  considered  to  project  growth  of  GDP  and  inflation. 

Mihajlo  Mesarovic  and  his  colleagues  (Mesarovic,  et  al.,  1970)  were  pioneers  in  multi-level 
modeling  of  complex  systems.  Their  conceptualization  of  the  task  of  multi-level  modeling  is 
useful: 

•  "Selection  of  strata,  in  terms  of  which  a  given  system  is  described,  depends  upon  the 
observer,  his  knowledge  and  interest  in  the  operation  of  the  system,  although  for  many 
systems,  there  are  some  strata  that  appear  as  natural  or  inherent."  (p.  40) 

•  "Contexts  in  which  the  operation  of  a  system  on  different  strata  is  described  are  not,  in 
general,  mutually  related;  the  principles  or  laws  used  to  characterize  the  system  on  any 
stratum  cannot  generally  be  derived  from  the  principles  used  on  other  strata."  (p.  41) 

•  "There  exists  an  asymmetrical  interdependence  between  the  functioning  of  a  system  on 
different  strata."  (p.  41)  -  any  stratum  depends  on  operations  of  lower  strata 

•  "Each  stratum  has  its  own  set  of  terms,  concepts,  and  principles."  (p.  41) 

•  "Understanding  of  a  system  increases  by  crossing  the  strata:  in  moving  down  the  hierarchy, 
one  obtains  a  more  detailed  explanation,  while  moving  up  in  the  hierarchy,  one  obtains  a 
deeper  understanding  of  its  significance."  (p.  42) 


FtepoitNo.  SERC-20l3-1R-a20-2 
J  une  6,  2013 

UNCLASSIFIED 

22 


Contract  Number:  H98230-08-D-0171 


TO  0029,  RT044a 


These  observations  are  highly  relevant  to  the  exposition  of  multi-level  models  of  complex 
systems  provided  in  this  report.  The  much  more  mathematical  material  in  their  treatise  is 
restricted  to  two-level  systems  and  assumes,  at  least  implicitly,  that  the  elements  on  each 
stratum  have  little  if  any  discretion  or,  at  the  very  least,  that  the  objectives  of  each  element 
(e.g.,  agent)  are  aligned  with  the  overall  system  objectives.  As  discussed  earlier,  this 
assumption  is  often  unwarranted  for  complex  socio-technical  systems. 

In  a  very  recent  report,  Mullen  (2013)  addresses  the  challenges  of  connecting  legacy  models,  at 
one  sitting  on  one  computing  platform,  to  meaningfully  address  new  questions  for  which  the 
component  models  were  not  inherently  created  to  answer.  It  is  much  easier  if  models  were 
designed  to  be  composable.  However,  integration  above  the  level  of  "plug  and  play"  can  still 
pose  significant  validity  problems. 

Zeigler  (2000)  addresses  integration  and  coordination  issues  in  multi-level  modeling.  He  is 
concerned  with  differential  equation,  difference  equation  (discrete  time)  and  discrete  event 
representations.  The  focus  is  on  how  to  computationally  integrate  these  representations. 
Rationalizing  and  integrating  differing  time  scales  is  a  dominant  issue.  This  is  a  necessary 
condition  for  meaningful  multi-level  modeling,  but  often  not  sufficient. 

Resolution  of  timing  issues  will  not  achieve  the  highest  levels  of  interoperability  articulated  by 
Tolk(2003): 

•  Level  4:  Common  Conceptual  Model/Semantic  Consistency 

•  Level  3:  Common  System  Approach/Open  Source  Code 

•  Level  2:  Use  of  Common  Reference  Models/Common  Ontology 

•  Level  1:  Documentation  of  Data  and  Interfaces 

•  Level  0:  Isolated  Systems 

The  difficulty  of  semantic  integration  is  readily  apparent  when  trying  to  integrate  financial 
spreadsheets  from  disparate  business  units  operating  in  different  markets.  However,  this  is 
easy  compared  to  Mullen's  challenge.  Assuring  semantic  integration  of  simulation  modules 
created  decades  apart  and  laced  with  undocumented  assumptions  is,  some  would  argue,  a 
fool's  quest.  Fortunately,  once  the  "plug  and  play"  requirement  is  relaxed  and  the  use  of 
arbitrarily  chosen  legacy  components  put  aside,  successful  multi-level  modeling  is  certainly 
possible. 


3.1  Problem  Structuring  Methods 

A  central  concern  is  appropriately  defining  the  problem  for  which  multi-level  modeling  is  to  be 
pursued.  Mingers  and  Rosenhead  (2003)  contrast  two  broad  classes  of  problems.  Well- 
structured  problems  are  those  "for  which  a  consensual  formulation  can  be  stated  in  terms  of 
performance  measure  or  measures,  constraints  and  the  relationship  through  which  action 
produces  consequences."  Unstructured  problems,  by  way  of  contrast,  are  those  "characterized 
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by  the  existence  of  multiple  actors,  multiple  perspectives,  incommensurable  and/or  conflicting 
interests,  important  intangibles,  and  key  uncertainties." 

These  authors  suggest  requirements  for  good  problem  structuring  methods: 

•  Enable  several  alternative  perspectives  to  be  brought  into  conjunction  with  each  other 

•  Problem  definitions  should  be  cognitively  accessible  by  actors  with  a  range  of 
backgrounds  and  without  specialist  training 

•  Operate  iteratively,  so  that  the  problem  representation  can  be  adjusted  to  the  state  and 
stage  of  the  discussion 

•  Permit  partial  or  local  improvements  to  be  identified  and  committed  to,  rather  than 
requiring  a  global  solution 

Mingers  and  Rosenhead  then  review  fourteen  methods  and  the  extent  to  which  they  satisfy 
these  requirements.  The  remainder  of  this  section  reviews  three  of  them:  Checkland's  Soft 
Systems  Methodology,  Beer's  Viable  Systems  Model,  and  Ulrich's  Critical  Systems  Heuristics. 

The  Soft  Systems  Methodology  (Checkland,  2003)  includes  seven  steps.  The  first  two  focus  on 
entering  the  problem  situation  and  expressing  it.  Next,  root  definitions  of  relevant  systems  are 
formulated.  Conceptual  models  of  human  activity  systems  are  then  constructed  from  the 
perspective  of  each  stakeholder.  These  models  are  compared  with  the  real  world  and  used  to 
define  changes  that  are  desirable  and  feasible.  Finally,  actions  are  taken  to  improve  the  real 
world  situation. 

Problem  definition  is  central  to  success  with  this  methodology.  Checkland  suggests  the 
following  guiding  questions: 

•  Clients  -  Who  are  the  beneficiaries  or  victims  of  this  particular  system? 

•  Actors  -  Who  are  responsible  for  implementing  this  system? 

•  Transformation  -  What  transformation  does  this  system  bring  about? 

•  Worldview  -  What  particular  worldview  justifies  the  existence  of  this  system? 

•  Owner  -  Who  has  the  authority  to  change  the  system  or  its  objectives? 

•  Constraints  -  Which  external  constraints  does  this  system  take  as  a  given? 

The  Viable  System  Model  (Beer,  1984)  is  premised  on  the  notion  that  all  organizational  systems 
are  composed  of  five  component  systems.  Problem  structuring  concerns  identifying  how  these 
five  systems  are  functioning,  or  not  functioning,  within  the  context  of  interest.  This  overall 
model  also  provides  guidance  for  designing  functions  that  may  not  yet  exist  in  this  context. 

•  System  1  in  a  viable  system  contains  several  primary  activities.  Each  System  1  primary 
activity  is  itself  a  viable  system  due  to  the  recursive  nature  of  systems  as  described  above. 
These  are  concerned  with  performing  a  function  that  implements  at  least  part  of  the  key 
transformation  of  the  organization. 
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•  System  2  represents  the  information  channels  and  bodies  that  allow  the  primary  activities  in 
System  1  to  communicate  between  each  other  and  which  allow  System  3  to  monitor  and 
co-ordinate  the  activities  within  System  1. 

•  System  3  represents  the  structures  and  controls  that  are  put  into  place  to  establish  the 
rules,  resources,  rights  and  responsibilities  of  System  1  and  to  provide  an  interface  with 
Systems  4/5. 

•  System  4  includes  the  bodies  that  make  up  System  4  are  responsible  for  looking  outwards  to 
the  environment  to  monitor  how  the  organization  needs  to  adapt  to  remain  viable. 

•  System  5  is  responsible  for  policy  decisions  within  the  organization  as  a  whole  to  balance 
demands  from  different  parts  of  the  organization  and  steer  the  organization  as  a  whole. 

Beer  is,  in  effect,  arguing  for  a  standard  multi-level  model  of  complex  organizational  systems. 
His  five  models  provide  a  template  for  problem  structuring.  The  extent  to  which  such 
standardization  is  meaningful  across  a  wide  range  of  contexts  is  discussed  later. 

Ulrich  (2003)  outlines  several  spheres  of  discourse.  They  range  from  local  to  multiple  domains 
to  public  to  societal.  These  domains  tend  to  have  differing  stakeholders  and  varying  interests. 
This  argues  for  a  multi-level  representation  of  the  overall  phenomena  of  interest.  His  Critical 
Systems  Heuristics  (Ulrich,  2003;  Jackson,  2003)  include  the  following  questions  for  guiding 
problem  structuring: 

•  Who  should  to  be  the  beneficiary  of  the  system? 

•  What  should  to  be  the  purpose  of  the  system? 

•  What  should  to  be  the  system's  measure  of  success? 

•  Who  should  to  be  the  decision  maker? 

•  What  elements  of  the  system  should  the  decision  maker  control? 

•  What  resources  and  conditions  should  to  be  part  of  the  system's  environment? 

•  Who  should  to  be  involved  as  the  designer  of  the  system? 

•  What  kind  of  expertise  should  contribute  to  the  design  of  the  system? 

•  Who  should  to  be  the  guarantor  of  the  system? 

•  Who  should  represent  the  concerns  of  those  affected  by  the  system? 

•  To  what  extent  should  those  affected  have  chances  of  relief  from  impacts? 

•  What  worldviews  of  those  involved  or  affected  should  influence  the  design 


3.2  Computational  Representations 

A  computational  formalism  is  a  modeling  formalism  with  a  well-established  computational 
implementation  or  implementations.  Computational  formalisms  operate  as  structured 
languages  for  representing  a  system.  As  such,  they  enforce  particular  ways  of  representing 
phenomena  that  provide  powerful  means  of  representing  certain  phenomena,  but  also  impose 
limitations  in  representing  others.  These  are  typically  stand-alone  means  of  representing 
systems  and  are  traditional  approaches  to  system  modeling  and  analysis.  Examples  include  the 
following. 
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•  Queueing  models  study  steady-state  behavior  of  systems  that  have  entities  that  engage  in 
transactions  with  a  set  of  resources  (originating  with  Erlang's  work  on  telecommunication 
systems).  Traditional  phenomena  of  interest  include  waiting  times,  throughput,  queue 
lengths,  etc.  For  the  most  part,  queuing  models  are  analytic  and  strongly  dependent  on 
assumptions  with  respect  to  probability  distributions,  etc.  There  are  some  computational 
approaches  that  relax  these  assumptions  and  use  numerical  methods. 

•  Discrete-event  simulation  was  developed  to  study  similar  phenomena  to  queueing  analysis, 
but  without  as  many  limiting  assumptions.  There  are  three  major  paradigms  or  worldviews 
for  discrete-event  simulation  (Kiviat,  1967).  The  process-interaction  paradigm,  in  particular, 
relies  on  a  network-of-queues  formalism.  Event-scheduling,  on  the  other  hand,  focuses 
primarily  on  events  in  terms  of  an  event  calendar  whereby  an  initial  event  set  executes, 
changing  the  system  state  and  scheduling  other  events  in  the  future.  Activity-scanning 
focuses  on  activities  and  the  necessary  pre-conditions  for  activity  initiation. 

•  Object-oriented  simulation  emerged  with  the  shift  to  object-oriented  programming  as  an 
alternative  to  representing  systems  as  a  network-of-queues.  The  idea  was  to  develop  class 
libraries  of  system  components  specialized  to  a  particular  domain  of  application  (Zeigler, 
1990). 

•  System  dynamics  was  established  to  study  complex  and  non-linear  phenomena  that  result 
when  system  components  affect  one  another  in  non-intuitive  ways  (Forrester,  1961).  Flows 
between  stocks  and  feedback  loops  are  important  concepts  for  representing  system 
phenomena. 

•  Agent-based  modeling  was  initiated  to  study  emergent  phenomena  that  results  from  the 
individual  behavior  of  a  networked  set  of  actors  (Holland,  1991). 

•  Optimization  models  seek  to  maximize  or  minimize  an  objective  function  subject  to  a 
constraint  set.  Algorithms  to  perform  the  optimization  may  either  be  exact  or  heuristic. 
Optimization  lends  itself  to  formal  descriptions  of  system  complexity. 

Each  of  the  above  can  be  used  to  specify  multi-level  models,  although  the  approach  would 
typically  be  ad-hoc  and  dependent  on  characteristics  of  the  system  being  modeled. 
Computational  formalisms  proved  to  be  poor  methods  for  communicating  models  for  a  variety 
of  reasons,  including  assumption  documentation,  stakeholder  understanding,  and  model 
maintenance  and  reuse.  Prose  is  not  well-suited  for  these  purposes  either,  due  to  its  ambiguity 
and  lack  of  formalism.  Thus,  interest  developed  in  visual  means  to  represent  systems  to  serve 
as  a  bridge  between  prose  and  computational  formalisms.  One  such  formalism  is  the 
system igram  (Blair,  Boardman  &  Sauser,  2007). 

Fundamentally,  systemigrams  are  a  conceptual  modeling  approach  using  soft  systems 
methodology.  They  are  used  to  convert  prose  to  a  visual  representation  for  purposes  of 
communication,  storyboarding  and  model  understanding  among  stakeholders.  The 
representation  emphasizes  concepts  of  emergence,  hierarchy,  boundaries  and  influence.  Major 
applications  include  systems-of-systems  and  networked  systems.  Potentially,  systemigrams 
could  be  used  to  represent  multi-level  systems  and  enterprises.  One  open  issue  is  how  to 
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transit  from  a  systemigram  to  a  computational  formalism.  A  potential  area  of  research  is  the 
characterization  of  templates  within  the  systemigram  representation  that  could  be  used  to 
derive  models  in  a  particular  formalism,  such  as  a  queueing  network  or  an  optimization  model. 

Another  visual  formalism  is  the  influence  diagram,  which  provides  a  method  for  representing 
probabilistic  events  and  decision  events  in  a  decision  tree  structure  (Howard  &  Matheson, 
2005).  The  main  motivation  here  is  to  provide  a  visual  means  for  model  communication  while 
maintaining  a  way  to  convert  the  representation  to  computational  form.  The  influence  diagram 
contains  a  sequence  of  time-step  nodes  that  are  formally  represented  via  a  combination  of 
chance  and  decision  nodes.  Potentially  multi-level  phenomena  can  be  represented,  for 
example,  by  expanding  a  decision  or  chance  node(s)  into  a  more  detailed  sub-model. 

Interest  has  continued  to  grow  for  visual  modeling  techniques  that  can  be  translated  easily  into 
computational  models.  Two  visual  formalisms,  IDEF  and  UML  in  particular,  have  made  a  major 
impact  on  modeling. 

IDEF  began  as  a  structured  way  to  represent  data  about  a  system's  or  organization's  inputs, 
decisions,  actions  and  activities,  independent  of  how  that  data  was  stored  (Mayer  et  al.,  1995). 
IDEF  evolved  into  different  generations  of  modeling  techniques.  IDEF  began  by  enforcing  a 
fundamentally  process-oriented  modeling  perspective.  But  subsequent  IDEF  specifications 
address  such  concepts  as  time-varying  system  behavior,  object-oriented  perspectives  and 
requirements  capture.  Each  of  these  concepts  lies  within  a  particular  specification,  and  there 
does  not  exist  an  integrated  modeling  framework  that  combines  them.  In  addition,  later 
specifications  tend  to  be  at  the  initial  specification  state.  The  set  of  specifications  is  shown  in 
Table  6. 

With  the  emergence  of  object-oriented  programming,  there  was  a  need  for  methods  to  specify, 
design  and  document  computer  programs.  A  variety  of  methods  were  developed  using  visual 
techniques  (Booch,  1991;  Rumbaugh  et  al.,  1990).  Eventually,  these  coalesced  into  the  Unified 
Modeling  Language  (UML)  (Jacobson,  Booch  &  Rumbaugh,  1999).  UML  enforces  a 
fundamentally  object-oriented  modeling  perspective  using  a  set  of  diagrams  for  software 
system  design  and  documentation.  The  two  major  types  of  diagrams  are  structure  diagrams, 
which  specify  the  components  of  the  software  system,  and  behavior  diagrams,  which  specify 
the  events  that  occur  during  execution.  Interaction  diagrams  are  a  subset  of  behavior  diagrams 
that  specify  flow  and  control.  UML  is  a  standard  adopted  and  managed  by  the  Object 
Management  Group  (OMG),  and  it  is  also  recognized  as  a  standard  by  the  International 
Organization  for  Standardization  (ISO). 
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Generation 

Purpose 

IDEFO 

Function  modeling 

IDEFl 

Information  modeling 

IDEFIX 

Data  modeling 

IDEF2 

Simulation  modeling  design 

IDEF3 

Process  specification  capture 

IDEF4 

Object-oriented  design 

IDEF5 

Ontology  description  capture 

IDEF6 

Design  rational  capture 

IDEF7 

Information  system  auditing 

IDEF8 

User  interface  modeling 

IDEF9 

Business  constraint  discovery 

IDEFIO 

Implementation  architecture  modeling 

IDEFll 

Information  artifact  modeling 

IDEF12 

Organization  modeling 

IDEF13 

Three  schema  mapping  design 

IDEF14 

Network  design 

Table  6.  IDEF  Versions 


With  the  success  of  the  UML  standard  for  software  systems,  there  emerged  a  strong  interest  in 
the  systems  engineering  community  to  have  a  similar  standard  for  systems  design,  analysis, 
verification  and  validation,  especially  given  the  wide  array  of  models  and  data  representations 
used  in  these  various  phases  of  the  system  lifecycle.  The  goal  was  to  provide  a  standard 
language  to  support  the  emerging  field  of  model-based  systems  engineering  (MBSE),  as  well  as 
a  set  of  resources  for  using  the  standard. 

The  OMG  commissioned  specification  of  SysML  as  an  extension  of  UML  to  support  systems 
engineering  (Friedenthal,  Moore  &  Steiner,  2008).  This  was  conducted  by  a  community-based 
effort  involving  many  individuals  and  organizations,  and  it  resulted  in  a  standard  for  SysML 
adopted  by  OMG. 

SysML  supports  both  object  modeling  and  process  modeling.  Both  types  of  models,  and  their 
integration,  are  important  to  multi-level,  enterprise  modeling.  SysML  provides  a  number  of 
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diagram  types  to  support  the  notion  of  modeling  enterprises  as  systems.  Diagram  types 
include: 

•  Structure  diagrams 

o  Block  definition  diagrams  to  model  system  structure 

o  Internal  block  diagram  to  represent  interfaces  and  interconnections  within  a  block 

•  Behavior  diagrams 

o  Activity  diagrams  to  model  state-based  behavior  from  the  perspective  of  inputs, 
outputs  and  controls 

o  Sequence  diagrams  to  model  sequences  of  events/messages  involving  different 
system  elements 

o  State  machine  diagrams  to  model  behavior  of  a  system  entity  from  the  perspective 
of  state  changes  caused  by  events 

o  Use  case  diagrams  to  represent  users  interacting  with  a  system  and  desired 
outcomes 

•  Requirements  diagrams  to  model  requirements  and  relationships  between  them  and  other 
system  elements 

•  Parametric  diagrams  to  represent  constraints  on  system  parameter  values 

•  Package  diagrams  to  organize  various  model  elements  (similar  to  UML) 

SysML  can  be  specialized  into  different  system  domains,  such  as  enterprises-as-systems.  As  an 
extension  of  UML,  it  can  provide  a  basis  for  software  design  (but  this  is  limited  since  many 
SysML  elements  are  not  found  in  UML).  Finally,  SysML  supports  the  notion  of  multi-level 
modeling  via  its  structural  and  behavioral  diagrams.  SysML  has  been  used  to  characterize 
organizational  and  enterprise  structure  and  behavior  for  purposes  of  knowledge  capture, 
sharing  and  reuse  (McGinnis  &  Thiers,  2012). 

While  SysML  has  many  obvious  advantages  for  multi-level  modeling,  it  has  limitations  especially 
in  application  to  socio-technical  systems.  SysML  typically  is  used  to  represent  the  technical 
aspects  of  systems.  The  main  exception  to  this  is  the  use-case  diagram,  used  to  represent 
stakeholder  use  of  the  system  to  meet  goals.  Behavioral  and  social  phenomena  can  be 
represented  in  a  technical  sense  via  state-machine  diagrams  and  such,  but  it  is  not  clear  to 
what  extent  this  is  adequate.  It  is  also  not  clear  whether  SysML  provides  much  support  for 
conflict  identification  and  resolution  among  requirements  from  a  diverse  set  of  stakeholders. 
This  is  a  critical  need  in  enterprise  modeling. 

A  variety  of  tools  exist  to  support  SysML  diagram  specification.  One  goal  of  the  MBSE 
community  is  to  use  SysML  as  a  specification  for  data  model  repositories  that  support 
population  of  different  analysis  models  for  specific  analyses.  This  is  an  active  area  of  research. 
With  the  limitations  of  single-formalism  modeling  approaches,  there  has  been  significant 
interest  in  combining  different  modeling  paradigms  within  one  formalism  or  technique.  One 
approach  that  seeks  to  combine  two  formalisms  is  the  work  of  Kim  et  al.  (2003)  in  combining 
IDEF  and  UML.  They  target  the  domain  of  enterprise  information  technology  systems,  which 
are  increasingly  ubiquitous.  These  systems  integrate  many  functions  and  have  stakeholders 
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with  different  perspectives.  There  is  no  modeling  methodology  or  language  that  supports  the 
entire  scope  of  designing  and  developing  such  systems.  Their  paper  explores  whether  different 
existing  tools  can  be  leveraged  to  provide  a  more  useful  and  powerful  approach  to  designing 
and  developing  enterprise  IT  systems. 

In  particular,  IDEF  appeals  to  "enterprise  modelers,"  who  are  engineers  who  design  business 
processes  and  engineer  IT  architectures  to  support  them.  UML,  on  the  other  hand,  appeals  to 
"distributed  object  system  modelers,"  who  design  and  develop  software.  This  work  is  useful 
because  it  demonstrates  how  data  elements  in  different  IDEF  representations  can  be  mapped 
and  made  consistent  with  data  elements  in  various  UML  representations,  as  well  as  leveraging 
prior  work  and  communities  of  interest. 

In  terms  of  application  to  multi-level  modeling  of  socio-technical  systems,  the  following 
questions  are  relevant. 

•  Can  this  approach  be  generalized  beyond  IT  systems  to  enterprises  in  general?  In  this  case, 
would  SysML  be  a  more  relevant  choice  than  UML?  Kim  and  colleagues  mentions  other 
methods/languages  such  as  UEML  and  CIMOSA.  Would  these  be  relevant? 

•  How  would  this  be  operationalized  in  modeling  tools? 

•  What  would  be  involved  in  scaling  this  up  if  other  methods/languages  are  needed? 

The  idea  of  multi-paradigm  approaches  and  toolsets  is  driving  much  of  the  current  effort  in 
model-based  systems  engineering,  since  the  broad  goal  is  to  support  system  design, 
development  and  testing  via  software  models  before  and/or  concurrently  with  those  same 
processes  using  physical  articles.  Such  a  multi-paradigm  approach  is  needed  to  support  these 
different  functional  activities  in  the  lifecycle.  A  variety  of  MBSE  techniques  are  supported  by 
software  tools  vendors  and  are  in  practice  in  a  variety  of  domains,  ranging  from  automotive  to 
aerospace.  Little  (2009)  describes  a  vision  for  MBSE  using  such  tools,  as  well  as  extensive 
application  types. 

One  critical,  emerging  technology  to  support  this  vision  is  that  of  model  transformation 
(Czarnecki  &  Helsen,  2006).  In  general,  model  transformation  is  the  concept  that  a  model  is 
input  into  a  procedure  that  outputs  a  new  type  of  model.  Model  transformation  is  a  concept 
from  the  OMG's  Model-Driven  Architecture. 

Here,  it  is  specifically  the  concept  that  a  descriptive  model  of  a  system  (e.g.,  in  SysML)  can  be 
used  to  generate  a  model  for  analysis  using  a  particular  computational  formalism  (e.g., 
simulation  or  optimization).  In  practice,  a  domain-specific  model  using  a  SysML  stereotype  is 
used  as  input  to  the  transformation,  which  then  outputs  the  structure  of  the  analysis  model 
within  its  formalism,  as  well  as  populates  the  analysis  model  with  data  (Batarseh  &  McGinnis, 
2012).  Model  transformation  is  an  emerging  field  of  research  that  is  increasingly  important  in 
MBSE. 
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One  of  the  appealing  features  of  model  transformation  is  that  a  single,  standard,  visual 
modeling  technique  such  as  SysML  can  be  used  to  store  the  system  model,  thus  serving  as  the 
communication  and  model  management  medium.  Typically,  SysML  is  specialized  for  the 
domain  under  consideration  for  this  purpose.  The  specific  analysis  models  are  generated 
automatically  using  a  transformation  technology,  alleviating  the  need  to  manage  multiple 
independent  models  across  many  different  analysis  paradigms. 

Model  transformation  is  in  the  early  stages  of  maturity.  However,  given  the  potential  multiple 
analysis  model  formalisms  that  may  be  used  within  a  multi-level  model,  it  could  be  of  potential 
use  in  model  and  data  management. 


3.3  Summary 

This  section  has  summarized  a  wide  range  of  material  including  a  description  of  the  "systems 
movement,"  a  summary  of  philosophical  underpinnings,  a  review  of  seminal  concepts,  an 
overview  of  complex  systems,  discussion  of  complex  adaptive  systems,  and  the  contrasts 
among  a  range  of  systems  approaches.  This  material  provides  the  building  blocks  for 
formulating  an  integrated  approach  to  multi-level  modeling  of  socio-technical  systems.  The 
"mortar"  between  these  building  blocks  will  be  seen  to  be  the  ways  in  which  human  behavioral 
and  social  phenomena  are  incorporated  into  the  overall  framework. 

4.0  Proposed  Framework 


Design  and  evaluation  of  complex  socio-technical  systems  can  be  addressed  using  the  multi¬ 
level  modeling  framework  shown  in  Figure  2.  This  framework  explicitly  represents  the  different 
levels  of  abstraction  underlying  system  behaviors  and  performance.  Note  that  this  framework 
is  a  derivative  of  the  ideas  of  Mesarovic,  Rasmussen,  Gharajedaghi,  Beer  and  Ulrich  discussed  in 
earlier  sections. 

The  levels  of  the  framework  can  embody  a  range  of  phenomena,  including  engineered, 
organizational,  and  natural  phenomena.  Of  particular  importance  to  socio-technical  systems 
are  human  behavioral  and  social  phenomena.  The  remainder  of  this  section  focuses  on  such 
phenomena.  The  "physics"  elements  of  multi-level  models  are  doubtlessly  important,  but  there 
are  many  sources  on  modeling  such  aspects  of  systems.  Hence,  this  section  focuses  on  the 
"human"  elements  of  multi-level  models. 

People  can  only  execute  work  practices  at  the  lowest  level  of  Figure  2.  Work  practices  are 
supported  by  delivery  operations  in  the  next  level,  which  only  exist  if  the  organizations  within 
the  system  structure  invest  in  and  sustain  these  capacities,  which  they  will  only  do  if  the 
domain  ecosystem  incentivizes  and  rewards  the  outcomes  of  these  investments. 
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Figure  2.  Multi-Level  Modeling  Framework 

The  domain  ecosystem  -  society  -  defines  the  objectives  for  the  system  and  the  rules  of  the 
game.  This  includes  explicit  or  implicit  specification  of  what  matters,  what  can  and  cannot  be 
done,  and  how  performance  is  rewarded.  These  specifications  incentivize  or  impede 
organizational  decisions. 

These  decisions  include  the  nature  of  the  system  capacities  considered,  levels  of  investment  in 
these  capacities,  and  assessment  of  subsequent  performance.  In  this  way,  delivery  operations 
are  created  and  sustained.  They  also  may  be  impeded  as,  for  example,  by  government  price 
controls  that  can  lead  to  disinvestment  in  capacities. 

Delivery  operations  provide  capacities  for  work.  These  capacities  can  include  engineered 
systems  (e.g.,  networks  and  databases,  devices  and  platforms),  processes  (e.g.,  procedures, 
plans),  and  venues  (e.g.,  factories,  playing  fields).  Work  practices  or  activities,  at  the  bottom  of 
Figure  2,  can  include  physical  manipulation  (e.g.,  lifting,  carrying,  controlling),  information 
provision  (e.g.,  informing,  advising)  or  social  interaction  (e.g.,  talking,  performing). 

The  four  levels  in  Figure  2  represent  different  levels  of  abstraction.  Within  each  level,  there  can 
also  be  levels  of  aggregation,  as  illustrated  by  Figure  3.  For  example,  individuals,  teams, 
specialties  (e.g.,  electricians)  or  whole  workforces  can  perform  work.  Processes  can  be  specific 
sets  of  steps,  generic  sequences  of  functions,  or  composite  procedures  for  all  automobiles  or 
patients.  Organizations  can  be  departments,  divisions,  subsidiaries  or  whole  corporations.  The 
"grain  sizes"  of  the  networks  at  each  level  reflect  the  level  of  aggregation  of  the  representation 
of  the  phenomena  at  that  level. 
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Figure  3.  Networks  of  Phenomena  at  Each  Level 

It  is  important  to  note  that  Figures  2  and  3  are  simplifications  for  the  sake  of  exposition.  Level¬ 
skipping  relationships  are  not  depicted.  Feedback  relationships,  that  are  often  pervasive,  are 
also  not  shown.  Finally,  "field"  types  of  relationships  -  e.g.,  gravity,  culture  -  are  also  not 
depicted.  Thus,  a  multi-level  model  of  a  realistically  complex  system  can  be  quite  a  bit  messier 
than  Figures  2  and  3  might  lead  one  to  imagine. 


4.1  Phenomena  &  Models 

There  can  be  a  range  of  socio-technical  phenomena  represented  in  Figures  2  and  3.  At  the 
People  level,  the  phenomena  of  interest  are  usually  human  behavior  and  performance  - 
individually,  in  teams,  or  in  groups.  Models  at  this  level  often  involve  input-output 
relationships  in  task  activities  with  the  focus  on  how  well  people  perform.  Not  surprisingly, 
performance  can  be  better  predicted  in  tasks  where  humans  have  little  discretion.  For  example, 
performance  in  landing  as  aircraft  is  more  predictable  than  performance  in  troubleshooting  an 
electronic  circuit,  which  is  more  predictable  than  responding  to  a  novel  emergency. 

For  the  Process  level,  human  decisions  concern  allocating  attention  to  the  capabilities  and 
information  needed  for  task  performance,  including  capabilities  and  information  from  other 
people.  In  this  regard,  a  central  socio-technical  phenomenon  are  the  social  networks  that 
enable  processes.  Of  course,  what  people  choose  to  attend  to  depends  on  the  choices 
available.  In  some  cases,  a  default  alternative  is  to  ignore  the  choices  altogether,  either 
intentionally  or  unintentionally. 
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The  Organization  level  is  typically  concerned  with  economic  decision  making,  drawing  upon 
classical  microeconomics  or,  more  recently,  behavioral  economics.  The  key  decisions  here 
include  allocating  resources  to  processes  and  assessing  the  current  and  projected  performance 
of  processes.  These  decisions  determine  what  choices  and  capacities  are  available  at  the 
process  level. 

At  the  Ecosystem  level,  policy  decisions  are  made  regarding  what  criteria  and  constraints  apply 
to  achieve  overall  objectives,  both  explicit  and  implicit,  drawing  upon  macroeconomics  and 
policy  sciences.  These  policies  tend  to  incentivize  or  inhibit  decisions  at  the  organization  level. 
Thus,  process  decisions  that  are  illegal  or  poor  investments  are  unlikely  to  be  made.  For 
example,  if  Medicare  will  not  pay  for  a  particular  procedure,  healthcare  providers  are  unlikely 
to  invest  in  providing  it. 

Another  example  of  socio-technical  phenomena  at  the  ecosystem  level  would  seem  to  be  the 
evolution  of  social  and  cultural  norms  and  beliefs  (Proctor,  Nof  &  Yih,  2012).  However,  these 
phenomena  tend  to  pervade  all  levels  and,  as  noted  above,  may  be  best  represented  as 
"fields,"  similar  to  gravity,  which  affect  the  parameters  at  all  other  levels.  For  example,  social 
systems  that  are  more  risk  averse  would  have  utility  functions  across  levels  with  different 
parameters  than  social  systems  where  risk  is  less  of  a  concern. 

There  is  a  rich  set  of  mathematical  and  computational  models  that  can  be  drawn  upon  to 
represent  the  range  of  phenomena  outlined  above.  Table  7  summarizes  a  range  of  alternatives. 
Development  of  a  multi-level  model  involves  choosing  from  these  and  other  classes  of 
representation,  creating  instantiations  particular  to  the  phenomena  of  interest,  and 
parameterizing  these  instantiations  based  on  data  from  the  domains  of  interest.  Thus, 
understanding  the  possible  choices  in  Table  7  is  very  much  a  first  step  in  formulating  multi-level 
models.  Instantiation  and  parameterization  involve  difficult  work  that  seldom  can  be  fully 
automated,  especially  if  one  is  concerned  with  Tolk's  semantic  interoperability. 

As  discussed  earlier,  translating  representations  to  computational  forms  is,  of  course,  a  critical 
step  to  developing  multi-level  models  of  complex  socio-technical  systems.  For  dynamic 
systems,  this  involves  defining  stocks,  flows,  feedback,  control,  and  error  measures;  choosing 
differential  or  difference  equations,  depending  on  whether  there  are  continuous  states,  or 
discrete  transitions;  and  deciding  how  to  compute  transient  responses  and  measures  of 
stability.  For  discrete-event  systems,  this  involves  defining  capacities,  flows,  queues,  resource 
allocations,  and  the  way  time  is  addressed;  choosing  arrival  and  service  processes  (e.g.,  Poisson, 
exponential)  and  possibly  characterizing  Markov  chains  with  discrete  states  and  continuous 
transitions;  as  well  as  deciding  how  to  compute  steady-state  responses.  For  agent-based 
systems,  one  must  define  the  rules  for  information  access  and  sampling,  decision  making,  and 
adaptation;  choosing  the  "grain  size"  of  agents  (e.g.,  individual  patients  versus  cohorts  of 
patients);  and  deciding  how  potential  emergent  behaviors  will  be  recognized.  For  situations 
where  optimization  makes  sense,  this  can  include  optimal  feedback  controls  (e.g.,  error  vs. 
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energy)  and  optimal  allocations  of  resources  (e.g.,  capacities,  resource  routes,  schedules, 
inventory). 


Level 

Issues 

Models 

Society 

GDP,  Supply/Demand,  Policy 

Macroeconomic 

Economic  Cycles 

System  Dynamics 

Intra-Firm  Relations,  Competition 

Network  Models 

Organizations 

Profit  Maximization 

Microeconomic 

Competition 

Game  Theory 

Investment 

DCF,  Options 

Processes 

People,  Material  Flow 

Discrete-Event  Models 

Process  Efficiency 

Learning  Models 

Workflow 

Network  Models 

People 

Consumer  Behavior 

Agent-Based  Models 

Risk  Aversion 

Utility  Models 

Perception  Progression 

Markov,  Bayes  Models 

Table  7.  Levels  of  Modeling  (Basole,  et  al.,  2011) 


Much  of  the  above  involves  modeling  and  representation  of  the  "physics"  of  the  environment, 
infrastructure,  vehicles,  etc.  These  are  certainly  important  elements  of  the  overall  multi-level 
model.  However,  the  greatest  challenge  in  developing  such  models  in  the  modeling  and 
representation  of  the  behavioral  and  social  behaviors  and  performance  throughout  the  system, 
especially  when  it  cannot  be  assumed  that  the  human  elements  of  the  systems  will  behave  in 
accordance  with  the  objectives  and  "rules  of  engagement"  of  the  overall  system. 


4.2  Human  Behavior  &  Performance  Modeling 

To  compile  the  possible  approaches  to  modeling  human  behavioral  and  social  phenomena  at 
the  various  levels  of  Figures  2  and  3,  the  tasks  of  humans  at  these  levels  must  first  be  defined. 
Table  8  provides  a  summary  of  these  tasks.  All  four  levels  include  both  strategic  and 
operational  tasks,  as  well  as  detection,  diagnosis  and  compensation  tasks. 
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At  the  Ecosystem  level,  the  key  strategic  task  is  to  set  the  "rules  of  the  game"  for  all  levels.  The 
operational  task  is  to  monitor  organizational  outcomes.  The  purpose  of  this  monitoring  is  to 
detect  anomalous  organizational  outcomes,  diagnose  the  causes  of  these  anomalies,  and 
decide  upon  appropriate  compensation  schemes.  All  of  these  tasks  are  likely  to  be  performed 
both  individually  and  in  groups.  Input-output  models  for  such  tasks  are  described  below. 

The  Organization  level's  strategic  task  is  to  allocate  resources  to  processes  so  as  to  optimize 
outcomes  within  the  rules  of  the  game.  The  operational  task  is  to  monitor  process  outcomes. 
The  purpose  of  this  monitoring  is  to  detect  anomalous  process  outcomes,  diagnose  the  causes 
of  these  anomalies,  and  decide  upon  appropriate  compensation  schemes.  All  of  these  tasks  are 
likely  to  be  performed  both  individually  and  in  groups.  Input-output  models  for  such  tasks  are 
described  below. 

The  strategic  task  at  the  Process  level  is  to  allocate  process  resources  to  enable  work.  The 
operational  task  is  to  monitor  work  outcomes  in  terms  of  performance.  The  purpose  of  this 
monitoring  is  to  detect  anomalous  work  outcomes,  diagnose  the  causes  of  these  anomalies, 
and  decide  upon  appropriate  compensation  schemes.  All  of  these  tasks  are  more  likely  to  be 
performed  individually  but  may  also  be  performed  by  groups  or,  more  likely,  teams.  Input- 
output  models  for  such  tasks  are  described  below. 

The  strategic  task  at  the  People  level  is  to  employ  process  resources  to  perform  work.  The 
operational  task  is  to  monitor  work  outcomes  in  terms  of  behaviors.  The  purpose  of  this 
monitoring  is  to  detect  anomalous  work  behaviors,  diagnose  the  causes  of  these  anomalies, 
and  decide  upon  appropriate  compensation  schemes.  All  of  these  tasks  are  more  likely  to  be 
performed  individually  but  may  also  be  performed  by  groups  or,  more  likely,  teams.  Input- 
output  models  for  such  tasks  are  described  below. 

The  concern  at  this  point  is  how  to  populate  multi-level  models  with  alternative  input-output 
relationships  for  the  twenty  cells  of  Table  8  for  different  contexts,  e.g.,  piloting  an  aircraft 
versus  managing  a  factory  versus  delivering  healthcare.  Fortunately,  there  is  a  rich  knowledge 
base  to  draw  upon,  including  National  Academy  studies  in  1998  and  2008;  frameworks  by 
Sheridan  (1974,  1992),  Rouse  (1980,  1983,  2007)  and  Rasmussen  (1986,  1994);  and 
computational  models  of  social  systems  (Carley,  2002,  2009). 

Pew  and  Mayor's  National  Academy  study  (1998)  reviewed  models  of  human  behavior  and 
performance  in  terms  of  attention  and  multi-tasking,  memory  and  learning,  human  decision 
making,  situation  awareness,  planning  and  behavior  moderators,  as  well  as  integrative 
architectures.  They  acknowledged  the  richness  of  this  knowledge  base,  but  also  questioned  the 
maturity  of  the  knowledge  relative  to  the  recognized  modeling  needs.  One  can  argue, 
however,  that  modeling  policy  decision  makers  behaviors  by  starting  with  how  they  visually 
read  characters  on  printed  pages  will  be  more  overwhelming  than  useful. 
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Enterprise 

Level 

Human  Tasks 

Strategic 

Task 

Operational 

Task 

Detection 

Task 

Diagnosis 

Task 

Compensation 

Task 

Ecosystem 

Set  Rules 

of  Game 

Monitor 

Organization 

Outcomes 

Detect 

Outcome 

Anomalies 

Diagnose 
Causes  of 

Anomalies 

Compensate 
for  Anomalous 

Outcomes 

Organization 

Allocate 

Process 

Resources 

Monitor 

Process 

Outcomes 

Detect 

Outcome 

Anomalies 

Diagnose 
Causes  of 

Anomalies 

Compensate 
for  Anomalous 

Outcomes 

Process 

Enable 

Work  With 

Resources 

Monitor  Work 

Outcome 

(Performance) 

Detect 

Outcome 

Anomalies 

Diagnose 
Causes  of 

Anomalies 

Compensate 
for  Anomalous 

Outcomes 

People 

Perform 

Work  With 

Resources 

Monitor  Work 

Activities 

(Behavior) 

Detect 

Outcome 

Anomalies 

Diagnose 
Causes  of 

Anomalies 

Compensate 
for  Anomalous 

Outcomes 

Table  8.  Human  Tasks  vs.  Enterprise  Level 


More  recently,  Zacharias  and  his  colleagues'  National  Academy  study  (2008)  considered  differ 
levels  of  models,  including  verbal  conceptual  models,  cultural  modeling,  macro-level  formal 
models  (e.g.,  systems  dynamics  models  and  organizational  modeling),  meso-level  formal 
models  (e.g.,  voting  and  social  choice  models,  social  network  models  and  agent-based 
modeling),  micro-level  formal  models(e.g.,  cognitive  architectures,  expert  systems  and  decision 
theory  and  game  theory,  and  interactive  games.  This  range  of  modeling  methods  and  tools  is 
much  more  relevant  to  the  phenomena  discussed  in  this  report  relative  to  the  earlier  Academy 
study. 

Sheridan  and  Ferrell's  (1974)  classic  on  human-machine  systems  addresses  modeling  of  a  wide 
range  of  human  behavior  and  performance.  Humans'  abilities  to  deal  with  uncertainty  are 
characterized  in  terms  of  probability  estimation,  Bayesian  probability  revision,  information 
measurement  and  channels,  information  transmission  tasks,  and  continuous  information 
channels.  They  provide  an  in-depth  review  of  manual  control  performance  including 
servomechanism  models,  input-output  identification  in  time  and  frequency  domains,  and 
optimal  control  models.  They  summarize  human  characteristics  in  terms  of  sensory  and 
neuromuscular  abilities,  as  well  as  intermittent  and  nonlinear  characteristics.  Human  decision 
making  and  utility,  decisions  under  risk,  signal  detection,  dynamic  decision  making,  and  formal 
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games  are  discussed.  The  knowledge  base  in  this  classic  is  quite  rich,  although  it  is  mostly 
focused  on  individual  behavior  and  performance. 

Sheridan  (1992)  addresses  supervisory  control  where  humans  interact  with  complex  systems 
via  computers  rather  than  directly,  which  is  quite  common  in  most  systems  now.  He  outlines 
the  generic  supervisory  control  functions  of  planning,  teaching  the  computer,  monitoring 
automatic  control,  intervening  to  update  instructions  or  assume  direct  control,  and  learning 
from  experience.  He  discusses  extensions  of  manual,  as  opposed  to  automatic,  control  theory 
beyond  his  earlier  treatment.  He  reviews  contemporary  results  on  human  attention  allocation 
models,  fuzzy  logic  models,  and  cognition  and  mental  models.  He  concludes  by  discussing 
limiting  factors  -  free  will,  ambiguity,  and  complexity  -  that  make  prediction  of  human  behavior 
and  performance  challenging. 

Rouse  (1980)  presents  a  wide  range  of  systems  engineering  models  of  human-machine 
interaction.  Estimation  theory  models  for  state  estimation,  parameter  estimation  and  failure 
detection  are  discussed.  Control  theory  models  for  manual  control;  quickening,  prediction  and 
preview  displays;  and  supervisory  control  are  reviewed.  Queuing  theory  models  of  visual 
sampling,  monitoring  behavior,  and  attention  allocation  are  illustrated.  Fuzzy  set  theory 
models  for  process  control  and  fault  diagnosis  are  discussed.  Finally,  artificial  intelligence 
models  are  presented  in  terms  of  production  systems,  pattern  recognition,  Markov  chains,  and 
planning  models.  Overall,  he  shows  how  the  "hard"  methods  of  system  dynamics  and  control, 
as  well  as  operations  research,  can  be  applied  to  modeling  human  behavior  and  performance. 

Rouse  (1983)  summarizes  a  wide  range  of  models  of  human  problem  solving  in  the  tasks  of 
failure  detection,  failure  diagnosis  and  failure  compensation  -  note  the  relevance  to  the  tasks  in 
Table  8.  He  reviews  eight  mathematical  models  of  failure  detection  and  eleven  mathematical 
models  of  failure  diagnosis.  The  key  conclusion  is  that  there  is  a  rich  base  of  computational 
models  to  draw  upon  for  modeling  human  behavior  and  performance  for  the  detection  and 
diagnosis  tasks  in  Table  8. 

Drawing  upon  a  wide  range  of  sources  (Rasmussen  &  Rouse,  1981),  Rouse  presents  a  general 
three-level  representation  of  human  problem  solving.  Rasmussen's  distinctions  among  skill- 
based,  rule-based,  and  knowledge-based  behaviors  (Rasmussen,  1981),  in  combination  with 
Newell  and  Simon's  (1972)  theory  of  human  problem  solving,  led  to  the  conclusion  that 
problem  solving  occurs  on  more  than  one  level  -  see  Table  9. 

When  humans  encounter  a  decision  making  or  problem  solving  situation,  they  first  consider 
available  information  on  the  state  of  the  system.  If  this  information  maps  to  a  familiar  pattern, 
whether  normal  or  abnormal,  they  perhaps  unconsciously  invoke  a  frame  (Minsky,  1975) 
associated  with  this  pattern.  This  enables  them  to  activate  scripts  (Schank  &  Abelson,  1977) 
that  enable  them  to  act,  perhaps  immediately,  via  symptomatic  rules  (S-Rules)  that  guide  their 
behaviors. 
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Decision 

State-Oriented  Response 

Structure-Oriented 

response 

Recognition  & 
Classification 

Frame  Available? 

Invoke  Frame 

Use  Analogy  and/  or  Basic 
Principles 

Planning 

Script  Available? 

Invoke  Script 

Formulate  Plan 

Execution  & 
Monitoring 

Pattern  Familiar? 

Apply  Appropriate  S-Rules 

Apply  Appropriate  T-Rules 

Table  9.  Problem  Solving  Decision  and  Responses 


If  the  observed  pattern  of  state  information  does  not  map  to  a  familiar  pattern,  humans  must 
resort  to  conscious  problem  solving  and  planning  (Johannsen  &  Rouse,  1983),  perhaps  via 
analogies  or  even  basic  principles.  Based  on  the  structure  of  the  problem,  which  typically 
involves  much  more  than  solely  observed  state  variables,  they  formulate  a  plan  of  action  and 
then  execute  the  plan  via  topographic  rules  (T-Rules).  As  this  process  proceeds,  they  may 
encounter  familiar  patterns  at  a  deeper  level  of  the  problem  and  revert  to  relevant  S-Rules. 

This  framework  has  important  implications  for  multi-level  modeling  of  complex  socio-technical 
systems.  Succinctly,  it  may  not  make  sense  to  represent  human  behavior  and  performance  for 
any  particular  task  in  Table  8  using  one  type  of  model.  Scripted  behaviors  may  be  reasonable 
for  familiar  and  frequent  instances  of  these  tasks.  However,  for  unfamiliar  and/or  infrequent 
instances  of  these  tasks,  a  more  robust  representation  is  likely  to  be  needed. 

More  recently.  Rouse  (2007)  presents  an  expanded  and  updated  set  of  the  foregoing  models, 
all  premised  on  the  nature  of  human  abilities  and  limitations  that  people  bring  to  their  tasks. 
He  elaborates  estimation,  queuing,  control  and  diagnosis  models.  He  also  provides  models  of 
human  behavior  and  performance  in  system  design,  information  seeking,  multi-stakeholder 
decision  making,  investment  decision  making,  strategic  management,  and  enterprise 
transformation.  The  overall  exposition  addresses  human  behavior  and  performance  in  tasks 
ranging  from  operation  and  maintenance  of  complex  systems  to  managing  enterprises  and 
leading  them  through  fundamental  change. 

Rasmussen  (1986,  1994)  discusses  a  range  of  models  for  attention  allocation,  signal  detection, 
manual  control  and  decision  making.  With  regard  to  decision  making,  he  addresses  human 
judgment,  decision  theory,  behavioral  decision  theory,  psychological  decision  theory,  social 
judgment  theory,  information  integration  theory,  attribution  theory,  fuzzy  set  theory,  scripts, 
plans  and  expert  systems,  and  problem  solving  models.  Rasmussen  also  presents  three 
important  conceptual  frameworks:  the  means-ends  abstraction  hierarchy,  levels  of  human 
control,  and  human  error  mechanisms.  These  frameworks  have  significantly  influenced  the  line 
of  reasoning  in  this  report. 

Carley  (2002,  2009)  addresses  computational  modeling  of  socio-technical  systems.  She 
represents  these  systems  as  "synthetic  agents  composed  of  other  complex,  computational  and 
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adaptive  agents  constrained  and  enabled  by  their  position  in  a  social  and  knowledge  web  of 
affiliations  linking  agents,  knowledge  and  tasks."  She  argues  that  the  capabilities  of  agents 
(cognitive,  communications,  information  seeking)  define  what  types  of  "social"  behaviors 
emerge,  and  concludes  that,  "The  use  of  computational  models  enables  generation  of 
meaningful  insights  and  the  evaluation  of  policies  and  technologies." 

Carley  and  Frantz  (2009)  discuss  a  set  of  computational  tools  for  simulation  of  social  systems. 
They  build  on  a  meta-matrix  representation  of  who  is  connected  to  who  and  the  nature  of  the 
connections.  Their  methods  and  tools  include  DyNetML,  a  universal  data  exchange  format  for 
social  network  data;  Automap,  a  software  tool  for  extracting  semantic  networks  and  meta¬ 
networks  from  raw,  free-flowing,  text-based  documents;  Organization  Risk  Analyzer,  a  software 
tool  that  computes  social  network,  dynamic  network,  and  link  analysis  metrics  on  single  and 
meta-network  data;  and  CONSTRUCT,  a  software  tool  that  provides  a  platform  to  supports 
virtual  experimentation  with  meta-matrix  data. 

Barjis,  (2011),  Carley  (2002b),  Cioffi-Revilla  (2010)  and  Dietz,  2006  discuss  a  variety  of 
methodological  considerations  related  to  the  characteristics  of  socio-technical  and  natural 
systems  and  defining  features  of  simulation  models.  They  elaborate  the  notions  of  enterprise 
ontologies,  enterprise  governance  and  enterprise  architecture.  This  material  represents 
recommended  ways  of  thinking  about  modeling  complex  socio-technical  systems  more  than 
presenting  models  per  se. 

5.0  Comparison  of  Domains 


Table  10  shows  how  the  multi-level  modeling  framework  can  be  applied  in  three  different 
domains:  healthcare  delivery,  energy  consumption,  urban  resilience,  and  military  operations. 
Application  of  the  framework  involves  representing  the  phenomena  at  each  level,  choosing 
models  to  represent  these  phenomena,  selecting  computational  means  to  operationalize  these 
models  across  levels,  including  the  flow  of  information,  e.g.,  on  incentives,  within  and  between 
levels  as  indicated  in  Figures  2  and  3.  All  of  these  components  provide  the  "engine"  for 
developing  interactive  visualizations  to  enable  exploration  of  alternative  system  designs  at 
multiple  levels,  e.g.,  process  designs  vs.  policy  rules. 

Considering  the  healthcare  delivery  example,  models  at  the  work  practices  level  would  include 
models  of  patient  disease  incidence  and  progression,  as  well  as  clinical  decision  making. 
Delivery  operations  would  be  modeled  as  process  flows,  including  information  flows.  Models  at 
the  system  structure  level  would  be  drawn  from  microeconomics  to  predict  organizations' 
process  investment  decisions.  The  domain  ecosystem  would  be  modeled  using  rule-based 
policies,  for  example,  for  alternative  payment  schemes. 
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Level 

Healthcare 

Delivery 

Energy 

Consumption 

Urban  Resilience 

Military 

Operations 

Domain 

Ecosystem 

Social  Priorities, 
Medicare/  Medicaid 

Public  Service 

Commission 

Regional,  State,  City 
Governance  System 

Military  Priorities, 

Rules  of  Engagement 

System 

Structure 

Providers,  Payers, 
Suppliers 

Utilities,  Builders, 

Contractors 

Mayor,  Council,  City 
Planning,  Emergency  Mgt 

Commanders,  Service 
Components 

Delivery 

Operations 

Care  Capabilities, 
Health  Information 

Generation,  Trans, 

&  Distribution 

Flows  Within  Delivery 
Infrastructures 

Strategies,  Tactics, 
Battle  Plan 

Work 

Practices 

Patient-Clinician 

Interactions 

End-User 

Consumption 

People  Consuming  Food, 
Water,  Energy,  Etc. 

Movement  of  Forces, 
Platforms,  Etc. 

Table  10.  Comparison  of  Domains 


A  multi-level  model  for  energy  consumption  would  be  populated  as  follows.  At  the  work 
practices  level,  consumers'  multi-attribute  utility  functions  would  describe  their  tradeoffs 
among  reduced  energy  bills,  investments  and  effort  involved,  and  value  attached  to 
contributing  to  environmental  sustainability.  These  models  would  likely  differ  among  different 
segments  of  the  population.  Social  network  models,  particularly  of  younger  households,  could 
portray  how  people  work  together  and  possibly  compete  to  save  the  most  energy.  At  the 
delivery  operations  level,  the  generation,  transmission  and  distribution  of  energy  would  be 
modeled,  with  variations  for  the  roles  that  renewables  such  as  solar,  wind,  and  waves  might 
play.  Systems  structure  models  would  include  microeconomics  model  of  the  key  stakeholders 
and  how  they  address  the  alternative  investments  in  this  arena,  including  how  utilities  address 
dynamic  pricing  within  the  rules  of  the  game  defined  at  the  domain  ecosystem  level  by  the 
public  service  commission. 

The  military  operations  example  is  representative  of  a  wide  range  of  multi-level  models  and 
simulations  developed  over  many  decades  (Mullen,  2013).  This  rich  legacy  provides  ample 
evidence  that  multi-level  models  and  simulations  are  eminently  feasible.  A  central  problem  has 
been  the  time  and  money  required  to  develop  them.  It  is  later  argued  that  the  foundation 
provided  by  this  report  can  improve  this  situation. 

The  resulting  interactive,  computational  model  can  be  termed  a  "policy  flight  simulator" 
(Rouse,  2013).  Such  simulators  can  provide  the  means  to  explore  a  wide  range  of  possibilities, 
thereby  enabling  the  early  discarding  of  bad  ideas  and  refinement  of  good  ones.  This  enables 
"driving  the  future  before  writing  the  check."  One  would  never  develop  and  deploy  an  airplane 
without  first  simulating  its  behavior  and  performance.  However,  this  happens  all  too  often  in 
organizational  decision  making  in  terms  of  policies,  strategies,  plans,  and  management 
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practices  that  are  rolled  out  with  little,  if  any,  consideration  of  higher-order  and  unintended 
consequences. 


6.0  An  Illustrative  Example:  Counterfeit  Parts 


6.1  Overview 

Potential  enterprise  modeling  problems  are  ubiquitous  in  government  and  industry.  To 
illustrate  our  multi-level  modeling  methodology,  we  select  a  case  study  involving  the  enterprise 
problem  of  counterfeit  parts  in  the  supply  chain  for  Department  of  Defense  (DoD)  systems. 
This  problem  has  gained  increasing  attention  and  concern  in  recent  years,  particularly  the  issue 
of  counterfeit  hardware  or  software  that  may  endanger  operational  performance  and  safety 
(GAO,  2010). 

Counterfeit  parts  fall  into  two  major  categories  -  parts  designed  with  malicious  intent  and 
those  designed  with  intent  to  defraud.  While  having  different  characteristics,  both  can  have 
serious  consequences.  The  former  may  escape  detection,  due  to  intent  to  provide  malicious 
functionality  under  certain  conditions  and  normal  operation  otherwise.  The  latter  may  fail  due 
to  quality  issues  after  an  initial  period  of  satisfactory  performance.  Fraudulent  counterfeits 
include  parts  re-marked  to  make  them  appear  as  original  equipment  manufacturer  (OEM)  or 
original  component  manufacturer  (OCM)  parts,  defective  parts  passed  off  as  functional,  or  parts 
scavenged  from  scrapped  assemblies  without  proper  documentation. 

Counterfeit  parts  have  been  documented  in  a  wide  array  of  defense  systems,  with  the  largest 
source  of  such  parts  coming  from  China  (Senate  Armed  Services  Committee,  2012).  Such  parts 
can  have  significant  adverse  effects  on  system  reliability,  resulting  in  increased  downstream 
costs. 

Counterfeit  parts  have  emerged  as  a  serious  issue  due  to  a  number  of  trends  in  defense 
systems,  as  well  as  the  global  economy. 

•  Modern  systems  are  increasingly  complex  in  terms  of  their  sheer  number  of 
components.  When  these  components  need  to  be  replaced  or  upgraded,  the  process  of 
ensuring  counterfeit  avoidance  is,  clearly,  more  difficult  than  would  be  the  case  if  there 
were  fewer  components. 

•  Systems  typically  are  being  used  in  service  for  much  longer  than  originally  anticipated. 

Thus,  replacement  parts  and  upgrades  must  be  provided  for  a  greater  than  anticipated 
time  frame.  A  well-known  phenomenon  in  the  defense  industry  is  that  of  supplier 
diminishment.  Over  time,  the  number  of  suppliers  for  a  particular  component 

diminishes,  as  the  original  equipment  manufacturer  or  original  component 


Contract  Number:  H98230-08-D-0171  TO  0029,  RT044a 

FtepoitNo.  SERC-20l3-1R-a20-2 
J  une  6,  2013 

UNCLASSIFIED 

42 


manufacturer  either  exits  the  market  or  goes  out  of  business,  or  as  new  technologies 
are  developed  requiring  new  replacement  parts  (AIA,  2011). 

•  At  the  same  time,  globalization  has  resulted  in  a  dramatically  increased  percentage  of 
electronic  components  made  overseas.  These  components  pose  the  most  risk  to  the 
DoD  supply  chain,  especially  as  these  suppliers  are  used  extensively  by  commercial 
supply  chains,  leaving  DoD  with  little  leverage  (AIA,  2011).  Foreign  suppliers  tend  to 
involve  more  risk  in  terms  of  providing  counterfeit  parts  than  domestic  ones. 

•  Concurrent  with  the  globalization  trend  are  two  trends  within  defense  acquisition 
programs  and  major  system  integrator  firms.  First,  major  defense  programs  are  seeking 
to  cost-share  acquisition  costs  with  other  governments  through  sales  of  systems  to 
those  governments.  The  inducement  used  for  this  is  to  locate  part  of  the  industrial  base 
for  a  particular  program  in  the  other  countries  whose  governments  are  would-be 
customers  (Kapstein,  2004).  Second,  prime  contractors  have  evolved  into  true  system 
integrators  that  assemble  major  sub-systems  produced  by  partner  firms,  as  opposed  to 
manufacturers  that  primarily  source  smaller  sub-assemblies  and  components  for  their 
assembly  operations  (Tang,  2009).  Thus,  the  prime  relinquishes  substantial  control  over 
its  supply  chain  by  decentralizing  design  and  production,  making  counterfeit  mitigation 
potentially  more  difficult. 

•  Use  of  the  internet  has  increased  the  anonymity  of  part  sources.  This  has  provided  a 
means  by  which  counterfeit  parts  can  more  easily  be  inserted  into  the  supply  chain.  For 
instance,  GAO  (2012b)  conducted  a  study  and  found  counterfeit  parts  were  available  on 
internet  purchasing  platforms  used  by  DoD. 

A  variety  of  different  counter-measures  are  available  to  address  counterfeit  parts.  In  the 
acquisition  phase  of  the  system  lifecycle,  these  include: 

•  Program  Protection  Plans,  which  articulate  measures  to  protect  security  of  a  program, 

•  Criticality  analysis,  which  indicates  what  sub-systems  and  components  are  mission- 
critical  and  thus  should  have  stricter  supply  chain  oversight  (e.g.,  through  sourcing  from 
trusted  suppliers), 

•  Software  assurance,  which  seeks  to  ensure  that  software  functions  as  intended,  free 
from  intentional  or  unintentional  defects, 

•  Robust  system  design  methods,  whereby  a  system  can  still  function  with  counterfeit 
components  or  potentially  experience  graceful  degradation,  and 

•  Trusted  system  design  methods,  whereby  a  system  detects  or  disallows  counterfeits. 

In  the  sustainment  phase  of  the  system  lifecycle,  counterfeit  counter-measures  include: 

•  Use  of  trusted  suppliers  for  replacement  parts  and  upgrades, 

•  Subsidy  of  OEMs  to  continue  making  replacement  parts  and  upgrades, 

•  Supply  chain  monitoring  (prevent,  detect,  respond), 

•  Incentives  to  primes  and  secondaries  to  monitor  their  sources. 
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•  Reporting  and  information-sharing  via  databases  set  up  for  counterfeit  incident 
documentation, 

•  Traceability  of  components  throughout  the  supply  chain, 

•  Penalties  for  counterfeiting,  and 

•  Intelligence  gathering  and  observations  on  defense  supply  chains. 

Many  of  these  options  are  included  in  official  DoD  policies  addressing  security  and 
counterfeiting  (DoD  2011;  DoD,  2012;  Kendall,  2012). 

Each  of  these  counter-measures  has  costs  associated  with  it.  For  instance,  robust  or  trusted 
system  design  methods  have  investment  costs  for  research  and  development,  and  then  likely 
operational  costs  for  systems  engineering  in  actual  programs.  Use  of  trusted  suppliers  may 
limit  supply,  causing  increased  part  costs  or  decreased  part  availability.  Supply  chain 
monitoring  incurs  inspection  and  other  costs.  Costs  may  be  borne  by  different  actors  in  the 
enterprise,  depending  on  the  particular  counter-measure.  Thus,  there  may  be  unanticipated 
behavioral  responses.  The  fundamental  question  is  which  portfolio  of  counter-measures  should 
receive  investments  to  mitigate  the  problem  of  counterfeits? 

It  should  be  noted  that  two  programs  exist  for  reporting  and  documenting  product  data  for 
government  procurement,  and  are  to  be  used  for  counterfeiting  incidents.  GIDEP 
(Government-Industry  Data  Exchange  Program)  is  to  be  used  for  reporting  by  industry.  PDREP 
(Product  Data  Reporting  and  Evaluation  Program)  is  to  be  used  by  government  agencies. 

Additional  information  on  the  issue  of  counterfeit  parts  can  be  found  in  (ABA,  2012;  Dept,  of 
Commerce,  2012;  GAO,  2011,  2012a;  Livingston,  2007a,  2007b;  McFadden  &  Arnold,  2010; 
Pecht  &  Tiku,  2006;  Stradley  &  Karraker,  2006). 


6.2  Enterprise  Problem  Characteristics 

Clearly,  this  is  an  enterprise  problem  due  to  the  multi-organizational  aspect  of  system 
acquisition  and  sustainment  practiced  by  DoD.  Organizations  involved  include  various  DoD 
agencies,  services  and  commands,  thousands  of  suppliers  organized  in  tiers,  plus  a  variety  of 
federal  agencies  with  jurisdiction  over  various  aspects  of  the  problem,  and  finally  Congress. 
Government  agencies  and  Congress  are  concerned  with  developing  and  implementing  laws  and 
policies  to  reduce  or  eliminate  the  risk  of  counterfeit  parts  via  testing  and  interdiction  or  use  of 
trusted/certified  suppliers.  A  variety  of  considerations  must  be  weighed,  some  of  which  affect 
the  entire  enterprise,  and  some  of  which  affect  individual  actors.  Cost  and  availability  of  spare 
parts  versus  safety  and  expected  performance  of  systems  serve  as  one  set  of  examples.  These 
issues  may  have  disproportionate  effects  on  certain  enterprise  actors,  depending  on  policies 
and  practices  used  to  address  the  problem.  Thus,  actors  are  incentivized  to  behave  in  certain 
ways,  some  intended  and  some  unintended  by  policy-makers. 
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As  an  enterprise  problem,  there  are  both  technical  and  socio  components.  Clearly,  the 
technical  aspects  of  the  problem  include: 

•  Systems  engineering  design  in  acquisition, 

•  Sustainment  networks  and  part  flows, 

•  Inventories, 

•  Inspection  regimens,  and 

•  Trusted  supplier  designation  based  on  objective  criteria. 

The  socio  components  include  the  following 

•  Trust  and  collaboration, 

•  Communication, 

•  Information-sharing,  and 

•  Reaction  to  incentives. 


6.3  Conceptual  Model 

Decisions,  actions  and  outcomes  may  occur  in  different  areas  of  the  enterprise,  ranging  from 
the  enterprise  level  to  the  level  of  a  single  actor  or  individual.  This  section  presents  a 
conceptual  model  of  this  breakdown  for  the  counterfeit  parts  case  study.  Table  11  shows  the 
important  constituent  elements  of  each  level. 


Level 

Elements 

Ecosystem 

U.S.  Government,  Department  of  Defense,  industrial  base,  economy, 
tax  base,  foreign  governments,  macro-trends 

System  structure 

Program  structure  (acquisition  and  sustainment),  integrated  product 
teams,  specific  customers,  supplier  network 

Delivery  operations 

Acquisition  processes  and  milestones,  sustainment  operations, 
design  facilities,  factories,  bases,  repair  depots,  inspection  facilities, 
delivery  channels 

Work  practices 

Systems  engineers,  program  managers,  sustainment  engineers, 
logisticians,  test  and  evaluation  personnel,  counterfeiters 

Table  11.  Multi-Level  Domain  Elements 


In  the  succeeding  sub-sections,  we  use  systemigrams  (Blair  et  al.,  2007)  to  illustrate  the 
relationships  between  different  components  of  the  overall  enterprise  model  both  between 
levels  and  within  levels.  The  intent  here  is  to  provide  an  overall  context  for  the  enterprise 
problem  that  can  be  used  to  develop  specific  models  to  answer  more  narrowly  focused 
questions  that  appeal  to  the  needs  of  specific  clients. 


FtepoitNo.  SERC-20l3-1R-a20-2 
J  une  6,  2013 

UNCLASSIFIED 

45 


Contract  Number:  H98230-08-D-0171 


TO  0029,  RT044a 


6.3.1  Domain  Ecosystem 


The  domain  ecosystem  for  the  counterfeit  parts  case  study  consists  of  the  Department  of 
Defense,  the  U.S.  government  and  relevant  security-related  agencies,  the  defense  industrial 
base,  the  overall  economy  and  tax  base  that  supports  defense  appropriations,  macro-trends 
that  impact  current  and  future  defense  programs,  and  policies  and  laws  that  govern  acquisition, 
sustainment  and  counterfeiting.  The  ecosystem  is  depicted  in  Figure  4. 

The  industrial  base  provides  platforms,  major  sub-systems,  sub-systems  and  components  for 
defense  systems.  The  industrial  base  is  influenced  by  macro-trends  such  as  globalization, 
outsourcing  and  off-shoring,  joint  ventures  with  foreign  governments,  and  new  business 
models  for  system  design  and  production.  Such  trends  may  expose  programs  in  the  ecosystem 
to  counterfeiting  risks  from  sources  that  have  either  strategic  or  economic  motivations. 

As  a  program  transitions  from  acquisition  to  sustainment,  its  industrial  base  shifts  from  design 
and  production  to  sustainment.  Sustainment  typically  operates  as  a  private-public  partnership, 
as  government  depots  play  a  substantial  role.  Such  concepts  as  performance-based  logistics 
come  into  play,  as  well,  whereby  a  prime  contractor  is  contracted  to  provide  a  certain 
performance  level  in  terms  of  metrics  such  as  system  availability.  Macro-trends  in  sustainment 
include  increased  system  life  spans  and  technology  advancements.  The  ecosystem  sees 
aggregate  outcomes  from  counterfeiting  in  terms  of  the  effect  on  overall  mission. 
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6.3.2  System  Structure 

The  system  structure,  shown  in  Figure  5,  level  focuses  on  the  various  organizations  (agencies 
and  firms)  that  interact  with  one  another  in  the  acquisition  and  sustainment  enterprise.  There 
is  typically  a  networked  structure  here,  with  some  amount  of  hierarchy. 

Any  particular  program  is  overseen  by  DoD's  Acquisition,  Technology  &  Logistics  office.  The 
industrial  base  provides  suppliers,  plus  the  prime  contractor,  for  a  program.  A  program's 
supply  chain  typically  is  organized  as  a  set  of  tiers  consisting  of  hundreds  or  even  thousands  of 
suppliers.  A  supplier  in  the  second  tier,  for  example,  provides  parts  to  suppliers  in  the  first  tier. 
This  tiered  organization  is  not  necessarily  hierarchical,  as  a  particular  firm  may  be  in  more  than 
one  tier.  In  addition,  a  firm  may  be  in  multiple  programs  and  may  collaborate  with  another  firm 
in  one  program  and  compete  with  the  same  firm  for  another  program's  contract. 

As  a  program  moves  from  acquisition  to  sustainment,  many  of  its  suppliers  will  continue  to 
supply  replacement  parts  for  use  at  different  sustainment  facilities.  However,  other  firms  from 
the  industrial  base  will  be  added  to  the  program  as  the  original  suppliers  may  elect  not  to 
continue,  or  not  be  able  to  continue.  Contractors  are  supposed  to  report  counterfeit  incidents 
to  GIDEP,  which  is  accessible  by  other  firms  for  supplier  monitoring.  Likewise,  government 
agencies  are  supposed  to  report  such  incidents  to  PDREP. 
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6.3.3  Delivery  Operations 

The  delivery  operations  level  focuses  on  the  various  processes  and  facilities  at  which  they  are 
performed  (Figure  6).  The  supply  network  is  cast  as  a  series  of  facilities  that  engage  in  design 
collaboration  between  the  prime  and  sub-contractors,  part  flows  that  eventually  result  in  major 
sub-systems  being  integrated  in  final  assembly  and  finally  cost  accruals.  These  activities  take 
place  in  the  context  of  acquisition  phases  as  spelled  out  in  DoD  5000.2.  Designs  and  articles 
pass  from  one  acquisition  phase  to  another  as  the  acquisition  matures.  The  program  office 
oversees  this  set  of  processes  and  communicates  and  enforces  various  regulations,  including 
counterfeiting  counter-measures. 

Systems  are  delivered  to  deployed  bases  from  the  production  processes  and  facilities.  Field 
maintenance  and  repair  is  done  at  these  facilities,  while  systems  are  sent  to  depots  or 
contractor  maintenance  and  repair  facilities  for  depot-level  work.  Parts  for  maintenance  and 
repair  are  held  in  inventories.  Inspection  procedures  may  be  used  there. 

While  the  figure  does  not  show  it  explicitly,  it  should  be  understood  that  the  supply  network 
facilities  evolve  over  time,  with  new  ones  being  added  and  current  ones  falling  out  of  the 
network,  as  a  program  moves  from  acquisition  and  production  to  sustainment.  It  is  here  that 
the  phenomenon  of  diminishing  supply  takes  hold. 
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6.3.4  Work  Practices 

The  work  practices  model  consists  of  the  individual  people  in  the  acquisition  and  sustainment 
enterprise,  as  well  as  the  work  that  they  perform  and  how  they  interact  with  one  another.  The 
emphasis  is  on  acquisition  and  sustainment  professionals  in  a  program  setting,  as  shown  in 
Figure  7.  A  program  manager  oversees  the  program  and  has  a  variety  of  government 
professionals  reporting  to  him  or  her  in  functional  roles. 

For  instance,  a  chief  systems  engineer  would  report  to  the  program  manager.  The  chief 
systems  engineer  may  have  other  engineers  reporting  to  him  or  her.  The  government  systems 
engineers  would  then  provide  oversight  for  the  systems  engineers  of  the  prime  contractor  to 
ensure  that  the  government's  interests  are  represented.  Typically,  the  systems  engineers  of 
the  contractor(s)  provide  most  of  the  work  for  system  design  and  development.  However, 
different  philosophies  of  division  of  labor  between  government  and  contractor  may  come  into 
play  over  time.  Thus,  the  model  should  provide  flexibility  in  this  regard.  The  systems  engineers 
of  the  prime  contractor  then  provide  oversight  for  the  systems  engineers  of  each  sub¬ 
contractor.  Usually,  the  government  systems  engineers  do  not  have  direct  oversight  or  contact 
with  those  of  the  sub-contractor  unless  it  is  through  the  prime. 
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Similarly,  there  is  government  oversight  of  the  prime  workforce  in  the  other  functional  areas, 
with  the  prime  then  providing  oversight  of  the  functional  areas  of  the  various  sub-contractors. 
The  workforce  performs  tasks  as  governed  by  DoD  5000.2  related  to  design,  development, 
testing,  maintenance,  repair,  etc.  To  address  specific  issues  that  overlap  functions,  workforce 
members  participate  in  integrated  product  teams  (IPTs).  There  are  likely  specific  counterfeiting 
counter-measures  done  at  the  individual  level  (e.g.,  adherence  to  guidelines  or  testing 
regimens)  called  out  explicitly. 

The  overall  workforce  is  affected  by  several  phenomena  such  as  training  (skills  and  skill  levels), 
social  networks  (cooperation  among  individuals),  collaboration  (cooperation  between 
functions)  and  trust.  Social  networks  can  be  shown  in  much  more  detail  as  relationships 
between  individuals  within  a  program,  whereas  trust  tends  to  be  more  of  a  field  effect. 


6.3.5  Relationships  between  Levels 

Each  level  has  relationships  with  the  other  levels,  as  shown  in  Figure  8.  For  instance,  the 
ecosystem  provides  the  incentive  structure  (e.g.,  contract  types,  penalties  for  counterfeiting. 
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available  funding)  and  policies  downward,  while  it  receives  performance  information  (cost, 
mission  effects)  from  below.  Figure  8  also  shows  the  typical  relationships  between  elements 
within  each  level  on  the  left  side  of  the  figure. 


Figure  8.  Relationships  between  Levels 

Each  level  has  performance  metrics  associated  with  it,  as  well.  The  ecosystem,  for  instance, 
focuses  on  mission-level  and  industrial  base  performance.  The  work  practices  level,  on  the 
other  hand,  focuses  on  individual  performance  in  terms  of  decision  quality  and  work  product, 
as  well  as  collaboration  performance  (e.g.,  between  functions  such  as  systems  engineering  and 
logistics).  The  performance  metrics  are  shown  in  Table  12. 
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Level 

Performance  Metrics 

Ecosystem 

Mission-level  performance 

-  Effect  of  counterfeiting  on  mission  achievement 

Industrial  base-level  performance 

-  Percentage  of  disqualified  firms  per  segment 

-  Percentage  of  OEM  and  trusted  supplier  firms  used  per  segment 

-  Number  of  single-source  firms  (OEM,  trusted,  other) 

Program  portfolio-level  performance 

-  DoD-wide  cost  of  counterfeit  interdiction/remediation 

System  structure 

Firm-level  performance 

-  Extra  costs  associated  with  counterfeit  prevention/remediation 

-  Lost  contracts  due  to  counterfeiting  problems 

Program-level  performance 

-  Extra  cost  of  counterfeit  prevention/remediation 

-  Extra  cost  due  to  counterfeit  discoveries  (e.g.,  extra  repair) 

-  Fleet  availability  cost  due  to  failures 

Delivery  operations 

Operation-level  performance 

-  Number  of  counterfeits  detected  via  failures 

-  Number  of  counterfeits  detected  via  inspection 

-  Inspection  cost  per  facility/process 

-  Failure  cost  (downtime,  repair,  etc.) 

Work  practices 

Individual-level  performance 

-  Decision  quality 

-  Work  product  quality 

-  Counterfeits  discovered/undiscovered 

Collaboration  performance 

-  Joint  functional  collaboration 

Table  12.  Performance  Metrics 


6.4  Analysis  Scenarios 

This  section  discusses  four  scenarios  around  which  specific  models  can  be  developed  for 
analysis.  These  scenarios  are  presented  in  Table  13,  which  provides  a  short  description, 
motivating  questions  and  types  of  models  needed. 

The  intent  is  to  be  able  to  derive  these  models  in  a  principled  manner  from  the  case  study 
context  presented  in  previous  sub-sections.  This  is  a  substantial  research  challenge  and 
involves,  at  least  in  one  approach,  specifying  a  computational  framework  that  embodies  the 
contextual  model.  This  framework  could  then  serve  as  a  library  for  enterprise  models,  at  least 
in  domains  related  to  counterfeit  parts  in  the  DoD  supply  chain.  Such  a  framework  and  library 
should  have  a  methodology  for  managing  assumptions  made  in  specific  models  relative  to 
other  models  and  to  the  library  components  and  framework. 
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Scenario 

Questions 

Models 

A  major  foreign  supplier  is 
influenced  by  its  government 

to  introduce  malicious 

counterfeits  into  multiple 

programs 

•  What  counter-measures 

are  most  effective 
enterprise-wide  (across 
multiple  measures)? 

•  Was  strength  of  program 
adoption  of  counter¬ 
measures  a  factor  in 
program  avoidance  of 
adverse  effects? 

•  Ecosvstem  -  Combinations  of  counter-measures 
as  policies 

•  Svstem  structure  -  multiole  orogram  networks 
with  program  offices  and  tiered  suppliers,  micro- 
economic  models  of  firm  behavior,  game  theory 
interactions  between  actors 

•  Deliverv  operations  -  discrete-event  simulations 
of  program  progress,  part  deliveries  and  logistics, 
etc.,  incorporating  effectiveness  of  following 
policies 

•  Work  practices  -  agent-based  models  of 
individuals  and  skills  and  skill  levels 

A  second-tier  supplier  has 
one  of  its  test  processes 
compromised,  since  it  has 

determined  it  cannot 

produce  parts  cost- 
effectively  within  its  fixed 
price  contract 

•  Can  this  be  detected  given 
the  output  stream  of  parts 
and  the  downstream 
inspection  processes? 

•  What  incentives  should  be 
applied  to  the  prime  to 
avoid  this  situation? 

•  Ecosvstem  -  Incentive  alternatives  for  suppliers 

•  Svstem  structure  -  game  theorv  interaction 
models  between  supplier  and  its  customer  and 
program 

•  Deliverv  operations  -  discrete-event  simulations 
of  part  outputs  and  inspection  processes 

•  Work  practices  -  agent-based  models  of 
individuals  and  skills  and  skill  levels 

DoD  is  faced  with  investment 

decisions  involving  research 
into  trusted  systems  and 
robust  systems 

•  Where  should  these 

investments  be  made  and 
how  long  will  they  take  to 
yield  results? 

•  How  effective  will  they  be, 
given  that  the 
counterfeiters  may 
develop  new  methods? 

•  Ecosvstem  -  Available  funding  and  priorities, 
threats 

•  Svstem  structure  -  R&D  agencies  and 
contractors,  research  strategies  at  firm  level 

•  Deliverv  operations  -  discrete-event  models  of 
investments  into  staged  research  programs  and 
outputs  with  probabilistic  success  outcomes  due 
to  technical  failures  and  changing  threat  profiles 

•  Work  practices  -  agent-based  models  of 
individuals  and  skills  and  skill  levels 

DoD  is  initiating  a  new 
program.  The  traditional 
way  of  business  is  that  the 
prime  conducts  in-house 
design  &  manufacturing  and 
out-sources  components. 
Recent  years  have  seen 
outsourcing  of  major 
assemblies,  some  to  foreign 
suppliers.  Problems  have 

ensued  with  this  business 

model. 

•  Which  business  model 
should  be  adopted, 
especially  with  respect  to 
counterfeit  parts 
vulnerability? 

•  What  counter-measures 

should  be  enacted  to 
support  either  model? 

•  Ecosvstem  -  Macro-economic  models,  svstem 
dynamics  models  for  economic  cycles,  available 
counter-measure  policies 

•  Svstem  structure  -  collaborative  network  of 
partnered  firms  for  program  vs.  command-and- 
control  structure  for  program 

•  Deliverv  operations  -  discrete-event  simulations 
of  program  progress,  production  and  logistics 
reflecting  different  program  models 

•  Work  practices  -  agent-based  models  of 
individuals  and  skills  and  skill  levels 

Table  13.  Scenarios 
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The  intent  is  to  be  able  to  derive  these  models  in  a  principled  manner  from  the  case  study 
context  presented  in  previous  sub-sections.  This  is  a  substantial  research  challenge  and 
involves,  at  least  in  one  approach,  specifying  a  computational  framework  that  embodies  the 
contextual  model.  This  framework  could  then  serve  as  a  library  for  enterprise  models,  at  least 
in  domains  related  to  counterfeit  parts  in  the  DoD  supply  chain.  Such  a  framework  and  library 
should  have  a  methodology  for  managing  assumptions  made  in  specific  models  relative  to 
other  models  and  to  the  library  components  and  framework. 

The  next  section  expands  on  this  notion  of  a  computational  framework  and  related  topics  with 
directions  for  future  research  in  enterprise  and  socio-technical  modeling. 

7.0  Research  Issues 


This  report  presents  a  conceptual  framework  for  multi-level  modeling  of  complex  socio- 
technical  systems,  provides  linkages  to  the  historical  roots  and  technical  underpinnings  of  this 
framework,  and  outlines  a  catalog  of  component  models  for  populating  multi-level  models. 
Thus,  the  framework  rests  on  an  impressive  body  of  knowledge. 


However,  there  are  several  fundamental  issues  that  need  to  be  addressed  for  this  endeavor  to 
mature  and  be  widely  employed  by  systems  scientists  and  engineers.  The  issues  and  questions 
outlined  in  this  section  need  to  be  resolved  if  multi-level  modeling  is  to  move  beyond  the 
current  state  of  each  instantiation  of  this  approach  being  an  idiosyncratic  and  often  heuristic 
creation  by  modelers  who  are  unaware  of  the  foundation  upon  which  they  can  build. 


7.1  Decomposition 

The  starting  point  for  multi-level  modeling  is  the  decomposition  of  an  overall  phenomenon, 
e.g.,  healthcare  delivery,  into  component  phenomena  at  varying  levels  of  abstraction  and 
aggregation.  A  central  question  at  this  point  is  what  phenomena  belong  in  each  of  the 
multiple  levels?  Further,  how  does  the  representation  of  each  phenomenon  depend  on  its 
level?  This  can  be  addressed  in  part  by  considering  natural  part-whole  relationships.  However, 
the  question  remains  what  wholes  and  what  parts  are  needed  to  address  the  question  that 
motivated  the  modeling  initiative? 

As  an  aside,  it  is  important  to  emphasize  the  need  to  be  very  clear  at  the  outset  about  what 
questions  the  model  is  intended  to  address.  When  the  only  objective  is  to  develop  a  model, 
and  the  questions  to  addressed  remain  elusive  and  ambiguous,  the  natural  tendency  of 
modelers  is  to  include  everything  they  can  imagine  that  will  ever  be  needed.  The  result  tends 
to  be  an  unwieldy  composition  that  is  difficult  to  understand  and  maintain. 
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7.2  Mapping 

The  choice  of  which  phenomena  will  be  represented  at  each  level  leads  to  choices  of  how  to 
characterize  each  phenomenon.  The  catalogs  of  models  discussed  earlier  can  be  quite  helpful 
in  this  regard.  Each  representation  will  have  defined  input-output  variables.  The  next  question 
is  what  variables  cross  the  levels  between  representations?  Further,  what  transformations  are 
needed  to  connect  across  levels?  Basic  issues  here  include  units  of  measure,  coordinate 
systems,  and  time.  Zeigler  (2000)  addresses  these  issues  within  the  context  of  dynamic 
systems. 

It  is  important  to  note  that  resolution  of  these  basic  issues  is  necessary  but  not  sufficient  for 
assuring  Tolk's  (2003)  semantic  interoperability.  Being  able  to  connect  two  models  and  have 
them  jointly  compute  some  outputs  does  not  assure  that  these  outputs  are  valid  and 
meaningful.  The  issue  here  is  one  of  "assumption  management."  Are  the  assumptions  of  the 
two  or  more  interconnected  models  compatible?  This  is  straightforward  when  the  modelers 
are  the  creators  of  all  the  component  models,  but  far  from  easy  when  some  of  the  component 
models  are  legacy  software  codes. 


7.3  Scaling 

It  is  often  the  case  that  models  begin  with  only  a  small  number  of  simulated  agents  (e.g., 
patients)  or  only  a  fraction  of  the  entire  transportation  network  that  is  of  interest.  The 
intention  is  to  scale  up  such  smaller  models  to  address  the  whole  problem  of  interest  once 
experience  and  confidence  is  gained  with  the  initial  models.  Scaling  is  often  very  difficult  and 
results  in  large  unfathomable  models  that  compute  very  slowly.  The  modelers  can  lose  any 
intuitions  of  what  is  happening  in  the  scaling  process. 

The  first  question  is,  given  the  targeted  scale  of  the  modeling  effort,  what  should  be  the  unit  of 
scale  for  each  phenomenon?  A  related  question  is  by  what  quantum  does  each  unit  scale? 
Perhaps  millions  of  patients  are  better  simulated  as  cohorts  rather  than  individuals.  Perhaps 
the  flow  of  thousands  of  vehicles  should  not  start  with  the  dynamics  of  each  vehicle,  but 
instead  consider  waves  of  vehicles. 


7.4  Approximation 

The  creation  of  large  multi-level  models  inevitably  requires  using  approximations.  The  central 
question  is  what  means  are  best  used  for  data  and  computational  efficiencies?  For  example, 
what  probability  distributions  are  used  for  arrival  and  service  times  in  a  discrete  event 
simulation?  Triangular  distributions  might  be  much  easier  to  implement  than  lognormal 
distributions,  compute  much  faster  once  implemented  and,  very  importantly,  have  parameters 
that  are  much  easier  to  estimate. 
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One  needs  to  ask  about  the  implications  of  different  choices?  For  example,  defining  agents  as 
cohorts  of  patients  rather  than  individual  patients  will  reduce  the  variability  across  patients.  If 
this  variability  is  the  primary  issue  of  importance,  some  countermeasure  for  the  variance 
reduction  may  be  needed.  In  general,  small  scale  examples  can  often  be  used  to  gain 
understanding  of  how  the  effects  of  approximations  propagate. 


7.5  Identification 

The  question  of  interest  here  is  how  can  structural  properties  of  processes  be  inferred  from 
design  and  operational  data  sets?  This  is  important  because  many  complex  systems  have  no 
"as  is"  blueprints,  i.e.,  such  systems  emerged  rather  than  being  designed.  Instead,  one  may 
have  data  on  millions  of  transactions  throughout  the  system's  processes.  One  needs  to  have 
algorithms  that  can  infer  processes  from  such  data  sets,  often  without  any  baseline  process 
maps  to  help  with  validation. 

This  raises  the  question  of  what  are  the  best  metrics  for  characterizing  the  "fit"  of  an  inferred 
process  to  a  data  set?  For  identifying  input-output  relationships,  one  could  use  mean-squared- 
error  as  the  metric.  However,  for  process  maps  where  relationships  among  nodes  are  the 
concern,  one  is  fitting  networks  to  data  rather  than  equations.  In  this  case,  one  might  use 
something  like  percentage  of  empirical  relationships  captured  by  the  network  representation. 


7.6  Parameterization 

Structural  representations  of  processes  will  usually  have  parameters  such  as  rates,  means,  and 
probabilities  that  need  to  be  estimated.  In  order  to  estimate  these  parameters,  one  first  has  to 
address  the  question  of  how  data  sets  can  be  accessed  and  normalized  across  elements  of  the 
enterprise.  For  example,  within  healthcare,  how  can  clinical,  financial  and  claims  data  sets  be 
combined,  while  maintaining  patients'  identities  and  rationalizing  varying  time  scales? 

Once  data  sets  are  combined  and  rationalized,  how  can  unbiased  parameter  estimates  best  be 
obtained  from  the  integrated  data  set?  A  key  issue  is  assuring  that  the  data  set  used  for 
estimating  parameters  is  representative  of  the  population  for  which  predictions  are  sought.  A 
more  refined  concern  is  estimating  parameters  for  baseline  "as  is"  systems  versus  potential  "to 
be"  systems. 


7.7  Propagation 

Structural  and  parametric  uncertainties  can  have  far-reaching  effects  as  they  propagate  across 
representations  and  levels  of  the  overall  model.  This  raises  the  question  of  how  uncertainties 
can  best  be  propagated  across  multiple  representations  at  multiple  levels?  In  particular,  how  is 
the  variability  associated  with  one  level  propagated  to  other  levels  when  simple  propagation  of 
point  estimates  is  unwarranted? 
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Various  mechanisms  might  be  adopted,  but  how  are  levels  of  variability  attenuated  or 
accentuated  by  different  approaches  to  propagation?  The  key  issue  is  that  approximations  can 
have  effects  beyond  the  immediate  impacts  motivating  the  approximations.  This  can  be  a 
rather  complicated  issue  and  have  significant  higher-order  effects  and  unintended 
consequences. 


7.8  Visualization 

If  multi-level  models  are  to  be  used  to  support  a  wide  range  of  decision  makers,  the  model 
outputs  have  to  be  accessible  by  people  who  are  far  from  modeling  and  simulation  experts. 
This  raises  the  question  of  how  the  "state"  of  a  multi-level  system  can  best  be  characterized 
and  portrayed.  The  answer  to  this  question  should  be  determined  by  the  nature  of 
visualizations  most  meaningful  to  the  key  stakeholders  in  regard  to  the  questions  targeted  via 
the  multi-level  model. 

Beyond  portraying  the  state  of  the  system,  stakeholders  are  often  concerned  with  the  nature  of 
relationships  between  levels  of  the  model.  How  can  the  relationships  within  a  multi-level 
system  best  be  portrayed  to  enable  experimentation  and  insights?  This  question  concerns  how 
best  to  enable  stakeholders  to  manipulate  the  relationships  between  levels  of  the  overall 
model.  Once  stakeholders  are  "in  the  loop"  of  choosing  assumptions  and  manipulating 
parameters,  stakeholder  buy-in  is  usually  greatly  enhanced. 


7.9  CURATION 

Section  4.0  outlined  a  wealth  of  component  models  for  potential  inclusion  in  multi-level 
models.  However,  this  wealth  is  not  often  used.  This  is  due  to  both  a  lack  of  knowledge  of 
these  resources  and  difficulty  in  accessing  them  (Rouse  &  Boff,  1998).  Professionals  in 
modeling  and  simulation  seldom  access  the  academic  journals  originally  reporting  these 
models.  Even  when  practitioners  are  aware  of  such  publications,  they  are  seeking  computer 
codes,  not  research  treatises. 

How  can  component  models  be  represented,  archived,  maintained,  and  accessed  to  facilitate 
rapid  model  integration?  Put  simply,  these  resources  need  to  be  curated.  There  needs  to  be 
one  point  of  access  for  the  many  hundreds  of  models  discussed  in  Section  4.0.  This  access 
should  enable  downloading  computer  codes,  documentation  on  assumptions  and  use,  and 
original  reports  of  the  development  and  validation  of  these  models.  Of  course,  this  begs  the 
basic  question  of  how  participating  organizations  can  be  incentivized  to  contribute  to  and  make 
use  of  the  curated  archive? 
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8.0  Conclusions 


Complex  engineered  and  natural  systems  can  be  characterized  as  complex  adaptive  systems 
where  independent,  yet  interactive,  intelligent  agents  pursue  their  goals,  often  in  conflict  with 
other  agents,  and  learn  and  adapt  to  the  changing  ecosystem.  A  multi-level  approach  to 
computationally  modeling  the  functioning  of  such  socio-technical  systems  can  provide  the 
means  to  understanding  and  then  transforming  such  systems. 

This  report  has  presented  a  conceptual  framework  for  multi-level  modeling  of  complex  socio- 
technical  systems,  provided  linkages  to  the  historical  roots  and  technical  underpinnings  of  this 
framework.  It  has  also  outlined  a  catalog  of  component  models  for  populating  multi-level 
models.  A  proposed  framework  was  given  for  multi-level  modeling  of  socio-technical  systems, 
including  discussion  of  the  phenomena  typically  associated  with  each  level,  as  well  as  wide 
range  of  models  of  human  behavior  and  performance.  A  comparison  of  multi-level 
representations  within  different  domains  and  an  illustrative  in-depth  example  were  presented. 
Finally,  fundamental  research  issues  underlying  multi-level  modeling  of  complex  systems  were 
summarized. 

There  are  significant  benefits  of  adopting  the  approach  advocated  in  this  report.  First  and 
foremost,  the  results  will  be  better  models  and  simulations  of  complex  socio-technical  systems. 
Behavioral  and  social  phenomena  are  often  central  to  the  questions  motivating  development  of 
models,  yet  these  phenomena  are  often  approached  in  very  rudimentary  ways.  Much  richer 
representations  are  certainly  possible. 

Multi-level  modeling  with  associated  interactive  visualizations  make  models  and  simulations 
much  more  accessible  to  a  wider  audience  of  key  stakeholders,  such  as  senior  decision  makers. 
The  abilities  to  interactively  explore  alternative  strategies,  policies  and  plans  enable  broader 
and  faster  consideration  of  alternatives.  The  result  tends  to  be  greater  "buy  in"  to  emerging 
and  eventual  decisions  (Rouse,  2013). 

Finally,  as  progress  is  made  on  the  issues  discussed  in  Section  7.0,  these  benefits  of  multi-level 
modeling  with  be  realized  faster  and  more  cheaply.  Multi-level  models  and  simulations  are 
much  more  likely  to  be  employed  when  it  does  not  take  many  months  or  even  years,  and 
hundreds  of  thousands  or  even  millions  of  dollars,  to  obtain  these  capabilities.  This  report 
provides  the  foundation  for  gaining  the  benefits  outlined  in  this  section. 
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